AngularCompiler marks not related files as affected in incremental build #21591

Which @angular/* package(s) are the source of the bug?

compiler-cli

Is this a regression?

No

Description

Any change in main.ts/m1.ts/m2.ts eg changing console.log(1) to console.log(2) will cause src/lib/types.ts to be included in affectedFiles (checked here https://github.com/angular/angular-cli/blob/master/packages/ngtools/webpack/src/ivy/plugin.ts#L569)

Looks like type only file is not included in ignoreForEmit Set nor included in emitted files so IncrementalCompilation can't find it previously emitted files and marks it as affected every time

Please provide a link to a minimal reproduction of the bug

https://github.com/SebastianPodgajny/ng42543/tree/affected-type-repro

Please provide the exception or error you saw

No response

Please provide the environment you discovered this bug in

Angular CLI: 12.2.1
Node: 14.16.1
Package Manager: yarn 1.21.1
OS: darwin x64

Angular: 12.2.1
... animations, cli, common, compiler, compiler-cli, core, forms
... platform-browser, platform-browser-dynamic, router

Package                         Version
---------------------------------------------------------
@angular-devkit/architect       0.1202.1
@angular-devkit/build-angular   12.2.1
@angular-devkit/core            12.2.1
@angular-devkit/schematics      12.2.1
@schematics/angular             12.2.1
rxjs                            6.6.7
typescript                      4.3.5

Anything else?

This is followup to this issue angular/angular#42543 which was partially resolved by JoostK PR angular/angular#42759