Repository navigation
Suggestion: CompilerHost should be able to "plug into" a LanguageServiceHost #9017
Description
Background
The traditional method of transpiling a typescript application is through ts.CompilerHost. This is awesome, and there are slowly becoming more and more CompilerHost based "flavors" of tsc that developers are wanting to use. I spoke about this with Alex Eagle (@alexeagle) and Mohamed Hegazy (@mhegazy) briefly on this as well:
Webpack Loaders
A webpack loader in its most simple form does the following:
function takeASingleFileThatIsServedToThisFunctionViaWebpack(rawFileStringContent) {
let newlyUpdatedRawFileString = rawFileStringContent + 'console.log(WOW THIS MAKES SENSE);'
return newlyUpdatedRawFileString;
}As you can see all a loader does is take a raw source, and return it. On top of this webpack handles file resolution through some crafty regex:
loaders: [
{
test: /\.ts/, // this regex string tells webpack which files to 'send' to the loader
loader: ‘ts-loader’ // this specifies the loader to use (resolves to node modules)
}
]
Webpack Typescript Loaders Cannot & Should Not Use ts.CompilerHost
So knowing what we know about how a file is tranformed and resolved through webpack, ts.CompilerHost is not an option for a ts-webpack loader. Why? Loaders are a powerful tool in webpack because they can be chained together with other loaders. For example: This Angular2 TS Loader I wrote. CompilerHost (as you know probably) runs a build/transpiliation against a whole project, and not one file per time. Therefore if you tried to plop a CompilerHost in a webpack loader, it would lose the context between each file.
Webpack TS Loaders & ts.LanguageServiceHost.
Because of this, the current typescript webpack loaders that are being used (ts-loader, awesome-typescript-loader) are both using ts.LanguageServiceHost. Why? Because LSG's support transpilation at a long-term lifetime one-file-at-a-time transpilation, persisting the transpile context back to the ts.Program. (This is tranditionally useful for IDE'S yes, but its a match made in heaven for a webpack loader). I would call this an 'edge-case' request, however, with 5+ million downloads a month, webpack is increasingly more popular than any bundler at the moment.
The Problem:
Trying to use ts.CompilerHost implementations in a ts.LanguageServiceHost is a huge pain, and doesn't port well into an existing LSH. Its true that they share some functions, however there lacks a way to use a CompilerHost in a LanguageService incrementally.
The Solution.
Create some way to attach the functionality of a custom CompilerHost implementation, and then perform it on a per-file basis, updating the LanguageServiceHost to help persist the transpilation/program contex. This will allow not only Stanislav Panferov (@s-panferov) and TypeStrong (@TypeStrong) able to create plugin's systems that allow for super powerful TS builds in webpack. Or instead a plugin system where you add/override certain functionalities to an existing LSH.
Please refer to this conversation thread of me explaining the issue to Daniel Rosenwasser (@DanielRosenwasser) with lots of examples:
Start: https://gitter.im/Microsoft/TypeScript?at=5757ab04662b042b7e596b70
End: https://gitter.im/Microsoft/TypeScript?at=5757b77445cf128e5f1d7429
References
#6508 (comment)
angular/angular#8759
TypeStrong/ts-loader#223
https://github.com/TypeStrong/ts-loader/issues/
s-panferov/awesome-typescript-loader#153
TypeStrong/ts-loader#224
Activity
DanielRosenwasser commentedon Jun 8, 2016
Thanks for this awesome writeup and summarizing the points of the conversation! 💯
TheLarkInn commentedon Jun 8, 2016
No problem I'm glad I could clarify!!!
mhegazy commentedon Jun 8, 2016
CC Chuck Jazdzewski (@chuckjaz) as we had a chat with about this issue.
So why do you use
LanguageServiceHostto do the loading. i would expect you just useCompilerHostall the time, given that there are no language service operations needed here.TheLarkInn commentedon Jun 8, 2016
CC Stanislav Panferov (@s-panferov) James Brantly (@jbrantly), please chime up if I'm not explaining this right.
Mohamed Hegazy (@mhegazy) Chuck Jazdzewski (@chuckjaz) so I said above that webpack is responsible for serving individual files one at a time to a loader. In that case, you can't use CompilerHost on one TS file at a time (because you lose program context between transpilation).
LSH provides the ability to build up your Program of files, and then additionally update them only compiling that one file in the context of the Program. Is there something I am missing?
aluanhaddad commentedon Jun 13, 2016
This may be relevant to plugin-typescript as well.
cc Frank Wallis (@frankwallis)
mhegazy commentedon Jun 14, 2016
The LanguageService API (LS), and the matching host, LanguageServiceHost (LSH), are built for IDE/Editor scenarios. The LS is mainly designed to be able to answer questions like get completions at position, quick info, formatting edits, etc.. one thing that the LS has to do well is update the state after an edit.
The Compiler API, (i.e. createProgram), and the matching host implementation, CompilerHost, are designed for generating outputs and errors given a set of sources.
Obviously there is an overlap. the LS is a higher level abstraction than the Compiler API. but, assuming i understand your needs correctly, you do not really need it.
If i understand correctly, what you need is given some source file, generate js/map/declarations, and compute errors.
And once you built that state, you want to keep it in memory and reuse it if one or more files change, and quickly regenerate errors and outputs.
if this is not an accurate representation of your scenario please let me know.
generating outputs and errors are done through
program.emitFilesandprogram.get*Diagnostics.Caching state, and reusing it is done through two things:
For reusing the SourceFiles, the CompilerHost API has
getSourceFilemethod specifically for this. it allows you to chose either to parse it, i.e. callcreateSourceFileor get it from a cache, say a previous Program object that has parsed this file, e.g.Program.getSourceFile(filename). This is what the--watchimplementation does. see https://github.com/Microsoft/TypeScript/blob/master/src/compiler/tsc.ts#L460-L473.The second is achieved by passing the old program to the
createProgramcall, see https://github.com/Microsoft/TypeScript/blob/master/src/compiler/program.ts#L1049.The LS can do this for you as well, as part of other things, but as noted before, you do not need the LS to do this.
I think the request here is for an API, that given a Program instance, and a set of edits, generate a new program with reusing as much as possible. if this is accurate, then we can definitely do that.
TheLarkInn commentedon Jun 14, 2016
First, I want to thank you and Daniel Rosenwasser (@DanielRosenwasser) for the support while you guys are in the midst of a crazy busy and awesome TS2.0 timeline. I apoligize if this response is messy but its hard for me to track everything in both loaders to ensure that the right things are done here.
Not quite, what's needed is: (again Stanislav Panferov (@s-panferov) and James Brantly (@jbrantly) please correct me on this)
That given a Program instance, and a single edit, generate the following:
languageService.getEmitOutput(filepath)outputFilesdependencyGraphfor the transpiled file (I believe this islanguageService.dependencyGraph[filePath])https://github.com/TypeStrong/ts-loader/blob/master/index.ts#L587-L633
s-panferov commentedon Jun 14, 2016
awesome-typescript-loaderright now has two basic scenarios of work (for watches):declarationenabled.1.1 Update program (pass all changed files)
1.2 Emit changed file: transpile source, get source map, get declaration emit. Uses program.emit
1.3. Extract dependency graph, pass it to webpack.
1.4 Check that full program is correct.
declarationdisabled andisolatedModulesenabled.2.1 Transpile changed files: transpiled source, get source map. Uses transpileModule, because it's much faster.
2.2. Extract dependency graph, pass it to webpack.
2.3 Update program and check that full program is correct.
awesome-typescript-loaderalso can do type-checking it a separate process, when all module resolution data and source files are send via IPC to the process which runs type checking and gets program diagnostics.s-panferov commentedon Jun 14, 2016
To simplify requirements:
isolatedModules, so we can take benefits from it. But sometimes we need to generatedeclarations.watchcycle. Here we already know our new files state.Sean Larkin (@TheLarkInn) wants some kind of plugins, that are able to modify source code using semantic information (if I'm correct). So maybe we need a fast way to update Program's semantic information from the new file independently of diagnostics output (to get the new semantic state available for the plugin during a
transpilephase)TheLarkInn commentedon Jun 14, 2016
Stanislav Panferov (@s-panferov) Thank you so much for the much more accurate translation! Your support in this has been invaluable.
mhegazy commentedon Jun 14, 2016
The dependency graph is not something that is provide today by the LS or the Compiler API. so we can use this issue to track adding it.
mhegazy commentedon Jun 14, 2016
one thing to note, as
isolatedModulesandtranspileModulehas been mentioned here.There are two operations, 1. transpilation, and 2. typechecking
transpilation can be done most of the time on a single file. you can use
transpileModuleto do that. but there are no semantic errors generated here, and no grantees about the correctness of the program.typechecking is a project wide operation. can not be done on a single file. but it does generate errors.
obviously, transpiling a single file is significantly faster than typechecking the whole project. but the two are not the same.
so if you do not care about errors, e.g. you report them from a different channel e.g. IDE, full build step, then use
transpileModule. if you do care about the errors,transpileModuleis not the correct API.erikbarke commentedon Oct 11, 2016
Mohamed Hegazy (@mhegazy): your explanation of LanguageService vs CompilerHost, caching etc should be in the Compiler Api docs :)
TheLarkInn commentedon Apr 14, 2017
Daniel Rosenwasser (@DanielRosenwasser) wanted to bump this issue for relevance. This is still a definite need for not only webpack loaders, but also angular and potentially broccoli-typescript plugin.
erikbarke commentedon Apr 15, 2017
Sean Larkin (@TheLarkInn), chiming in here - this is relevant to karma-typescript too since the
Karmatest runner serves its files to its preprocessors asynchronously one at a time. My current implementation collects the files sent to the preprocessor usinglodash.debounce, and then performs a deferred compilation of the whole project after a small delay.The upside of this approach is that I can use the
CompilerHostand keep state in a cachedProgrambetween compilations when runningKarmain watch mode, making updates incremental.The downside is that it's such a clunky workaround imo: it adds an unwanted delay to the compilation, there's always the risk of race conditions introducing subtle bugs etc. So this would be a more than welcome addition to the Typescript API.
mhegazy commentedon Feb 8, 2018
With #20234, i think the underlying issues in this thread should all be addressed. #20234 introduces two new concepts, a Builder and a Watcher.
A Builder can take an old program and a new one (after an edit), and answer questions like getDiagnostics and emit efficiently. it does that by caching the state from the old program and only recomputes things that needs to be computed, based on the dependency graph of the project.
A Watcher, can watch either a tsconfig.json or a set of input files. it caches most of the module resolution/file IO operations needed, and triggers the creation of a new program on edit.
combined together you can get the same output
tsc --watchand compiler-on-save in VS achieves today.We are working on docs at the moment, but here is the PR for the docs: microsoft/TypeScript-wiki#169. you can also find an example of the new API being used in TypeStrong/ts-loader#685