.NET Config Files configSource outside the application directory folder

I have two applications one a console application and the other an ASP.NET app. They both need to know the same appSettings and connectionStrings. So ideally I would like to use the configSource property of app.config/web.config files to point that to a central location. For example

<connectionStrings configSource="D:\connectionStrings.config"/><appSettings configSource="D:\appSettings.config"/>

That however fails with an error:

The configSource attribute is invalid.: The configSource 'D:\appSettings.config' is invalid. It must refer to a file in the same directory or in a subdirectory as the configuration file.

Is there anyway to still use the configuration managers appSettings/connectionStrings and get the values from an external location?
I'm happy with having to add code to do it, but I don't want to have to replace the entire configuration manager system.