Digging through a lot of files

I have a scenario where I need to scan a lot of xmlfiles from elmah-logs in the size of 300 000 files.

Powershell was a fun project for this but couldn't really do the job due to performance issues.
When reading 300 000 files using powershell like example below, the script ran for 368 minutes.


I tried running parallell jobs and using dotnet to read files instead but nothing could complete with using MS Logparser.
So enter Log Parser Studio.
This neat tool managed to comb through 300 000 files in 47 minutes instead!
It is a bit tricky to formulate the queries however. Heres an example of getting elmah logs where a variable named HTTP_REFERER contains a key value

select * FROM '[LOGFILEPATH]' where  string like '%http://www.mycompany.com/subsite%' and name like 'HTTP_REFERER'

So in conclusion for same set of 300 000 files
Powershell took 368 minutes
Log Parser Studio took 47 minutes

References:

https://www.microsoft.com/en-us/download/details.aspx?id=24659  - Log Parser 

https://gallery.technet.microsoft.com/office/Log-Parser-Studio-cd458765

https://blogs.technet.microsoft.com/exchange/2013/06/17/log-parser-studio-2-0-is-now-available/

Using hashtables to combine values

For a job I needed a good way of combining a primary value with optional subvalues in a xml-file.
Below is an example of how to extract attributes from a xmlelement into a hashtable and then join two hashtables into one. I couldn't find a method for extracting elements on the web, folks usually goes with subnodes instead of attributes. I'm partial to attributes and had to get creative, but the solution was quite simple. The Attributes from the element only show the actual elements from the xmlfile. Its not obvious when browsing the object that -name property can be lifted, but here it's used for creating a new hashtable-row



Samplescript for proof of concept


This eventually led to createwebsFromStructure.ps1 that uses powershell splatting to build commands with the resulting hashtable.

References:
https://powershell.org/2013/01/23/join-powershell-hash-tables/ - source for join-hashtable function
https://technet.microsoft.com/en-us/library/gg675931.aspx - source for details on splatting