Powershell

How To Grep in PowerShell

For those who have lived in the Linux/Unix command line, the ‘grep‘ command is a commonly used way of finding something that you want in a chunk of data.

Øyvind Kallstad did a great writeup of comparing a bunch of ways to use PowerShell instead of grep which is worth reading.

The article covers a bunch of scenarios, and is centered around starting with the ‘grep’ command and working with it. However, there’s the other common use case of running a different command, then piping those results to grep to search for something.

This blogpost was triggered by Janet who asked me this fair question:

As with poor cute cats, there’s more than one way to skin PowerShell.

I had to do some research and asking around on this, because normally I’d filter out the property of the object I was looking at, and work with that. Using the get-process example:

get-process | where ProcessName -like "*foo*

That works, but it’s still a lot clunkier than what a grep user would expect. An easier way would be to use the ‘findstr‘ program (which also has a bunch of useful swtiches):

get-process | findstr foo

I say program because ‘findstr’ is not a PowerShell cmdlet, but it’s still native to Windows and works perfectly fine. It’s case sensitive though, so you need to use -i for case insensitive results.

That’s great for simple stuff, but we’re sort of breaking what PowerShell does. You’re no longer dealing with a standard PowerShell object, so further piping and processing won’t really work.

The ‘proper’ PowerShell way would be to use the ‘Where-Object’ command:

Get-Process | Where-Object {$_ | Select-String "foo"}

A bit longer, but you can shorten ‘Where-Object’ to ‘Where’. Although more involved, it’s good to get into the habit of doing it this way, so when you’re piping this to the next command, it still says as a standard object that can be read and manipulated.’

(Update 24th Feb 2017) As Steve_N points out in the comments section, there’s a much shorter way of doing this:

ps *foo*

That’s it. Many PowerShell commands have inbuilt aliases, including ‘get-process’. You can see what this is with the command ‘get-alias -definition get-process’

This shows that ‘gps’ and ‘ps’ are both aliases to the command  ‘get-process’. You can also create your own aliases with the ‘set-alias‘ command.

The ‘*foo*’ part works because the command assumes the -name switch has been used, which lets you define what criteria to search and show in the ProcessName field. This is the same way that many commands don’t need the -identity switch used, because it’s written to assume you’re going to tell it what identity/username/upn to work with.

This can also be piped to something else, so it’s a winner. It’s less ideal for scripts though, because it’s much harder to read, and you can’t assume that everyone will know the short alias of a full command.

(Update ends)

PowerShell isn’t a Linux/Unix command line, but Microsoft have incorporated many of the concepts from bash. If you still can’t bear to use PowerShell on Windows, there’s always the Linux Bash Shell on Windows.

Thanks again to Steve Mclnerey for the grep advice 🙂

PowerShell – ‘While’ Loop Statement

There’s a lot of different ways to loop commands in PowerShell, and here’s one I just learnt (thanks Nathan Kewley for spending the time talking me through this!):

Scenario: You create a brand new user in Active Directory, but need to wait for things to sync before you make a change to the user. If you want to automate these steps, you want to check that the user exists before running more commands against it.

Answer: The ‘While‘ statement. This lets you loop a command ‘while’ something is a certain value. For example, you may want a script to loop for two minutes, or until a certain value is true or false.

With my script below, it will check if the value $running is nothing (null), which it is because we just made it up. Because it’s true, it’ll then continue on to do whatever is in the curly brackets. Here, I’m running a command the enable a user in Skype for Business, but also setting the result of that as the variable $running.

If the command works, $running now has a value of the created user, so as it loops again to see if $running is null, it won’t be, and the ‘while’ statement is done.

If the command fails however, and shows the dangerous red warning around the user not existing, nothing gets set to the $running variable. That means, when it loops again, $running will still be null so it’ll try again and again and again.

while($running -eq $null){
 $running = Enable-CsUser -Identity testuser -SipAddress testuser@contoso.com
}

That’s rather dangerous of course, what if it’s forever $null? It’ll run forever, so we’d better put in some failsafes.

while($running -eq $null){
 if($CheckUser -le '10'){
  $CheckUser++
  start-sleep -s 10
  $running = Enable-CsUser -Identity testuser -SipAddress testuser@contoso.com
}
}

OK, this time we’re doing a couple more things. We’ve got two curly bracketed things to run now, the first is an ‘If’. If $CheckUser is less or equal to 10, then do the next curly bracket thing. The first time this runs that value again doesn’t exist because we just made it up, and nothing is less or equal to 10. The If statement is true, so it moves onto the next segment.

The $CheckUser++ command just adds ‘1’ to the value of $CheckUser – starting off at null or 0, so will turn into 1. As the statement loops, that number will increment all the way up to 11. Once it’s 11, the If statement is no longer true, so bombs out.

We’ve also added the start-sleep command, which is just a 10 second wait before doing anything. If we didn’t have that there, the 11 loops before it fails would be over incredibly quickly.

The last thing we can add is an event to occur once the ‘If’ statement is no longer true:

while($running -eq $null){
 if($CheckUser -le '10'){
  $CheckUser++
  start-sleep -s 10
  $running = Enable-CsUser -Identity testuser -SipAddress testuser@contoso.com
}else{
Throw "Unable to create SfB User"
}
}

All we’ve done here is added the ‘Else’ section, which only runs when the ‘If’ isn’t true. Once the $CheckUser variable hits 11, the ‘Else’ command runs and throws up an error, with the aptly named ‘Throw’ command.

Hopefully this is enough to explain the basics of the ‘While’ command.

Searching Multi-Valued Properties in PowerShell

I’ve been playing with Office 365 commands in PowerShell and had to do a search. Sounds simple, but depending on what you’re searching, some scenarios are less basic than others.

Everything in PowerShell is an object. Usually, a property in PowerShell has a single value, such as:

UserPrincipalName: afowler@contoso.com

which is one of the results from Get-MsolUser. However, another property is different:

AlternateEmailAddresses: {microsoft@contoso2.com}

Visually, the difference is just the {} braces that contain the value. These braces mean that the property has been built to contain multiple items, rather than a single item.

If I wanted to see a list of all UserPrincipalNames, I’d use this command:

Get-MsolUser -all | select UserPrincipalName

A nice list of UPNs would display on the screen. However, that same command against AlternateEmailAddress, all that comes up is a bunch of blank lines.

To make this work, we need to select the value and show all the expressions of each value:

get-msoluser -all | select @{Name=“AlternateEmailAddresses”;Expression={$_.AlternateEmailAddresses}}

To then search on those values with the ‘where’ command, you’d have to write it like this:

get-msoluser -all | select @{Name=“AlternateEmailAddresses”;Expression={$_.AlternateEmailAddresses}} | where {$_.AlternateEmailAddresses -like "*contoso*"}

The good news is, for a where search by itself, you can forget all that and go back to basics:

Get-MsolUser -all | Where AlternateEmailAddresses -like "*contoso*"

Because of this requirement on the Select command, it lead me down the wrong path for a bit. There’s other reading on how to list all the values of a multi-valued property

If you’re still lost and want to get started with PowerShell, try checking out this PowerShell Basics video

Changing Many File Permissions Without Inheritance

I ran into a scenario when moving files from an older Windows Server 2003 box to Windows Server 2012, where I couldn’t access folders even as an administrator.

It turned out that not having Users (ServerName\Users) causes problems beyond Windows Server 2003. When moving a mass of files with many folders lacking inheritance, this can be a problem.

After some research and testing, it’s reasonably easy to modify NTFS security permissions to lots of files, while leaving existing settings in tact and not requiring inheritance to apply changes.

Scripting Guy covered it pretty well here, but here’s the condensed version:

First, install File System Security PowerShell Module because it’s easier to do than using native PowerShell Set-ACL commands. This can be installed on a remote box from where the files are, but remote can be slow based on latency and the amount of files you’re dealing with. Read the installation notes on that page so you’re ready to go with the module.

You can test it’s working by running a command like:

get-ntfsaccess -path c:\PutAFilenameHere.Now

You’ll get the permissions of the file back.

To change your permissions on mass, you need to get a listing of the files and pipe that to your modified settings:

dir \\FileServerName\ShareName -recurse | add-ntfsaccess -account “BUILTIN\users” -accessrights read

Note that the -recurse switch gives you all files and subfolders of the share, and although the permission you’re looking at via Windows Explorer will show Users (ServerName\Users), this is actually the BUILTIN\users permission. If you try to use the servername, you’ll get this error:

Add-NTFSAccess : Cannot bind parameter ‘Account’. Cannot convert value “servername\users” to type
“Security2.IdentityReference2”. Error: “Some or all identity references could not be translated.”

That’s it, you can now add, remove or modify permissions all over the place without affecting other existing permissions or affecting the inheritance.

Edit: The actual issue I describe here can also be fixed by changing a few Group Policies too.

Softerra Adaxes – Several Months In

logo-1

Softerra Adaxes is an Active Directory management & automation tool which I’ve grown very fond of.

First I reviewed Softerra Adaxes, then I actually bought Softera Adaxes and even did a brief case study for them. I thought it would now be good to share how far we’ve come through using this tool, and what the experience is like for those considering this option of automation. Here’s my thought process and how I personally approached the rollout, along with my experiences along the way:

Initially to me, the idea of having an ‘Outlook rules’ style approach to building a system that automated user management was enough to me. We’d been creating accounts manually for a long time, and the process was documented but took 20 minutes or so to perform. There was also a lot of room for human error, especially when someone was interrupted while creating an account.

There was of course the ‘selfish’ reason of not wanting to do these user management tasks myself, but it’s hard to pass those tasks off with the inherit risks or lack of knowledge of the tools being used to ease the process. This is what had held me off writing my own giant PowerShell script to automate all the steps.

After mucking around with the Adaxes basics, I started to realise that this software solution seemed to actually deliver on what I was personally looking for – something that wasn’t complex, but also let me define whatever criteria, business rules and caveats to the user creation process that I wanted. On top of that, there was inbuilt webpages where I could deliver these options to other staff requiring no software installs, and the ability to show or lock down whatever I chose, to both control and protect the Active Directory environment.

It did take a few weeks to set up properly, but I wouldn’t have really spent more than an actual day’s worth of work in those few weeks doing it. That was just to create a new user in all the various systems I wanted, with our unique user setting requirements. I wouldn’t say the entire system is so simple and easy to navigate that you can get cracking, but it’s also not complex. Once you find the setting or understand how Adaxes achieves a solution, it’s not difficult to set things up.

The inbuilt functionality of website templates – where you can create multiple sites displaying whichever fields you like to whichever users you like – is a good way to deliver the solution to end users. You can have a page for IT and another page for Finance with completely separate functions to best fit each use case.

For me, it was great that I could create websites with zero programming requirements. It’s all driven by a GUI, and somehow it’s still very flexible in what it can do. It might be frustrating to someone who actually writes code, but that’s not who would normally be using this solution. I really feel it’s aimed at someone like me, the IT Pro/Sys Admin who wants to automate and allow others to use the tools, without needing to code or expect others to run PowerShell commands themselves.

adaxes1Basic site with one option – menu and right side options can even be hidden if required.

Once I’d finished the user creation process and published the method of doing so to a website, I had internal staff muck around with it and use it, purely for new user creations. The feedback I received was immediately positive – that 20 minute or so process had been reduced to a few minutes, and even generated out an email saying the account creation was done. This in itself to me was the tick of a successful project, and I knew I could do a lot more around automation and empowering others to do repeatable tasks.

Some of the problems I hit on the user creation automation were:

  • After upgrading from Lync 2010 to Skype for Business 2015, there were intermittent errors popping up for creating a SfB user. This was a known problem to Softerra, and took several months to resolve with a new version of Adaxes. I did have a workaround luckily, so it only took some rule modifying to work around it until a proper solution was found.
  • ‘User unknown’ – I ran into some problems where I’d create the user or enable them for Exchange, but then the next command wouldn’t find the account. Adaxes was faster than what other systems could replicate changes, so some tactful ‘start-sleep’ PowerShell command steps during the workflows to allow replication to occur before the next step triggered. This does mean that the overall process can take a minute or two, and the person who triggered the user creation has to wait for it to finish.
  • Not all functionality was available that I needed in the GUI. For example, creating a Skype for Business user is easy, but you can’t assign a policy. Instead you need to use PowerShell commands to do what you want. That took a bit longer and needed more testing, but wasn’t much of an issue once I found that out.
  • When a new user was created that already existed (e.g. another John Smith – john.smith) I hadn’t considered that scenario. I asked in the Adaxes forums and was told how to run some pre-checks to make sure the username and phone number were unique and bomb out if they weren’t, rather than half creating an account and having to clean it up afterwards.
  • The upgrade process isn’t painful when a new version of Adaxes comes out (which came out while I was doing the user creation and I wanted to try upgrading early on), but there’s a few more steps than next, next finish. An uninstall is required with backing up a few files, then a fresh install and importing what you backed up. I’m hoping that will be streamlined a bit in the future.

After the user creation process was settled, I started to create more automation tasks. Deprovisioning was an obvious one, and was a lot easier than user creation as well as taking a lot less time to set up. This command would clean up all the bits and pieces from an account, including home drives and Exchange settings (along with moving the mailbox to a different database). This was rolled out relatively quickly.

I should also note, the logging is very helpful. If someone triggers a command from the website, they can see if it was successful or not, or where it failed. It made testing easy to do, but I was also able to read through logs via the GUI on the server to find out more about what failed and why.

adaxes2Updating options on one of the web interfaces – no coding required.

I then decided to wait for common scenarios to come up and build them as needed. We often had ‘returning staff’ which if their Active Directory account still existed, I couldn’t use my user creation method when the account already exists. This took a rethink of how I’d designed my rules so far, and decided to re-do a lot of it in a more modular fashion. Because there’s the ability to copy and paste rules, this was a lot easier than I expected. The end result was that I’d have a list of modules to run against a task – e.,g. a new user would call commands such as ‘enable email’ and ‘enable Skype for Business’ which my new ‘returning staff’ would call ‘re-enable email’ but the same ‘enable Skype for Business’ command as a new user. This now meant I could move a mailbox from one database to another and unhide the user from the Global Address Book when they returned, but because all users have their Skype for Business disabled, that step was the same in either scenario.

Another valuable idea I had was to let users control the membership of Active Directory groups that they were the owner of. After some mucking around, I created a website solely for that purpose. The great part about it was that whomever logged onto the site (with passthrough authentication so no extra typing required) could only see groups they were an owner of, based on the Manager field in Active Directory. This gives anyone in the company who is in control of a group, the ability to add or remove members without any IT assistance required. Perfect for application owners who control who can get to their application or not via a security group.

My next task will be the automation of a user name change. With the updated modular design, I can copy out the steps that I need and modify them to my new requirements; of course finding the hour or two to build and test this is the hardest part. (Note: Between the week of writing this and publishing, I’ve now done it.

I’ll give praise to both the Adaxes forums and their helpdesk support via email- almost always, within 24 hours max (and usually 4-5 hours) I’d get a specific and clear answer on how to do something I couldn’t work out personally, and it was from someone who knew the product rather than a basic 1st level helpdesk type response.

I hope this gives a real impression of my experience and opinion of Softera Adaxes at a high level, after using it for an extended time. There’s no real gaps to the product that I’ve found. and you can pick and choose as to how much customisation you want to do through PowerShell scripting. I’m still happy with the product, and it will continue to evolve with us.

Windows Max Path Is Now A Lot Bigger

The legacy 8.3 filename restrictions that came from the old MS-DOS days are (for the most part) long gone, but one of the other lingering legacy limitations is the 260 character limit.

Microsoft have a great article about how all this works and the reasons why. With Windows 10 anniverasry edition and Windows Server 2016, it’s possibe to get around the 260 character limit with some caveats. The new limit is 32,767 characters!

When researching this, I found quite a few articles that said how to enable the setting but didn’t really go into it any deeper, and my testing found that it’s not as simple as described. Enabling “Long Paths” doesn’t magically remove the limit, it enables longer paths in certain situations.

Firstly – enabling the policy itself. There’s a mix of information out there, and I’ve found some catches.

One of the mentioned methods of turning the feature on is to use Group Policy at Computer Configuration > Administrative Templates > System > Filesystem > Enable NTFS long paths. This doesn’t exist on my fully updated Windows 10 Pro install:

paths3No ‘Enable NTFS longs paths’ option?

I also checked on Windows 10 Enterprise fully patched, and the option was also missing. I found this similar option though, one level down:

paths4‘Enable win32 long paths’ option

The name and description are very similar. I then found this technet thread which agreed that they are the same setting.

After applying the setting and rebooting, I tested via Command Prompt to see how far I’d get:

The filename or extension is too long

I wasn’t very successful. “12characters” contains… 12 characters, so 18 folders * 12 characters = 216. Then add the slashes and you’re around 233 characters. Not quite the 260 limit, but close. Why wasn’t it working?

A few reasons; the app itself needs to support the new API calls to go beyond 260 characters, and I dare say Command Prompt hasn’t been touched yet due to the potential of breaking things.

What about Windows Explorer? This is where things got a bit strange. I couldn’t create a folder in that same path for the same reason, so I created a share on the very bottom of the tree, went to the share name and started creating more folders. I then went back to the original path to see if I could navigate all the way to the bottom, and I could:

path5Lots of subfolders!

22 folders called ’12 characters’ = 264 characters by itself, and I was then able to create a subfolder called “New Folder”. What’s strange about this is that Windows Explorer itself wouldn’t break the 260 character wall directly, but once it was passed, it was happy to read through and continue on further.

Back on the command prompt, it had let me navigate one folder further than before into the 19th, but wouldn’t delve any deeper:

path2“The full path of 12characters is too long”

What about PowerShell? That seemed to be very happy with the extra characters, so I made a complex script containing the lines “md 12characters” and “cd 12characters” many, many times. PowerShell happily went mad creating subfolder after subfolder, although the speed of subfolder creation went from ludicrous speed to very very slow as it ran.

Seeing what Windows Explorer would do, I was surprised that I’d hit a different limit:

paths6No more expand option

The ability to drill down further had gone. I could see 29 folders in the tree, and the 30th on the right hand pane, but couldn’t actually get into it. I also couldn’t create folders or files at that level or seveal levels up:

paths7Too long.

Back into PowerShell, I had to scroll to see beyond my current folder path!

paths8Lots of folders

I was also able to create files at that level. From this, if you’re going to use long paths in Windows Server 2016 or Windows 10, use PowerShell to manage your files!

This to me seems a good reason for Microsoft to not make Long Paths on by default. It should only be used for special cases, and a lot of things may break or just not support it. For example, if you’re doing a file level backup, will your backup software both read and write beyond the 260 character limit?

The best use case I can think of currently is having a location you can extract out long paths (maybe that came from a Unix box?) so you can adjust back down to the 260 limit, or get out the files you need. Microsoft always has pressure to look after legacy, and I can see the 260 character limit being around for a very long time.

Null and Not Null with PowerShell

Finding out if an object has a null (i.e. blank) value or not isn’t a difficult task to do.

Consider this scenario – you’ve found a bunch of old disabled accounts that someone forgot to remove the ‘Manager’ field. Finding accounts that have another field that would be populated for a current employee but blank for a departed would be a reasonable way of finding the problem accounts, then you could null the ‘Manager field. (note – you could just refine your search to disabled accounts but that’s not as fun).

To find all Active Directory users that have a blank ‘Department’ field is easily done with this command:

get-aduser -filter * -properties department | where department -eq $null

Then, showing the users that don’t have a blank ‘Department’ field is a slight change. You can’t use !$null (!=not), but you can use -ne (not equals)

get-aduser -filter * -properties department | where department -ne $null

You can also check for users that have a manger by switching ‘department’ to ‘manager’:

get-aduser -filter * -properties maanger | where manager -ne $null

Easy. Adding in a second ‘where’ statement so we can get results of users that have a manager, but no department means we have to add in a few extra characters to make PowerShell happy:

get-aduser -filter * -properties department,manager | where {($_.department -eq $null) -and ($_.manager -ne $null)}

The results can be a bit hard to read, so piping (|) to a select command will just show us the results of each user we want to see:

get-aduser -filter * -properties department,manager | where {($_.department -eq $null) -and ($_.manager -ne $null)} | select name

Finally, to blank the ‘manager’ field, we can swap the ‘select name’ command with this:

get-aduser -filter * -properties department,manager | where {($_.department -eq $null) -and ($_.manager -ne $null)} |  set-aduser -manager $null

You can then go back to a previous command to confirm you get no results. As always, check your data first before blanking out a bunch of user’s values!

Update

As @mickesunkan pointed out, the above isn’t the most efficient way to do searches. I’m sure I’ve mentioned this before, but I’m not always going to write the cleanest, quickest way of doing something. For a once off tasks this really doesn’t matter. For a daily task it starts to matter – not really by itself, but if you keep making more and more inefficient scripts, you’re putting extra unnecessary load on your environment with lots of LDAP lookups.

Above, I’m just getting ALL AD users. You could use a better filter and narrow down to a certain OU. You could also put part of your ‘where’ command into the filter, such as this:

get-aduser -properties manager,department -filter {department -notlike “*”}

This doesn’t work for the ‘Manager’ field though, you’ll see this error:

get-aduser : Operator(s): The following: ”Eq’, ‘Ne” are the only operator(s) supported for searching on extended attribute: ‘Manager’.

I couldn’t work out a way of putting the $null value as part of the filter, but if you do – please share 🙂

 

@mickesunkan also wrote this github code showing a few differnet ways to do this search, and which way is most efficient. Thanks Micke!