Category Archives: Uncategorized

Installing Python Wheel files on an Azure App Service

Per Microsoft: Some packages may not install using pip when run on Azure. It may simply be that the package is not available on the Python Package Index. It could be that a compiler is required (a compiler is not available on the machine running the web app in Azure App Service).

Example, you may receive an error like this when trying to install a specific package (in this case, trying to install Pandas):

Command: "D:\home\site\deployments\tools\deploy.cmd"
Handling python deployment.
KuduSync.NET from: 'D:\home\site\repository' to: 'D:\home\site\wwwroot'
Copying file: 'requirements.txt'
Detected requirements.txt.  You can skip Python specific steps with a .skipPythonDeployment file.
Detecting Python runtime from runtime.txt
Detected python-2.7

Found compatible virtual environment.
Pip install requirements.
Downloading/unpacking Flask==0.12.1 (from -r requirements.txt (line 1))
Downloading/unpacking numpy==1.15.0rc2 (from -r requirements.txt (line 2))
Downloading/unpacking pandas==0.22.0 (from -r requirements.txt (line 3))
  Running setup.py (path:D:\home\site\wwwroot\env\build\pandas\setup.py) egg_info for package pandas

    Could not locate executable g77
    Could not locate executable f77
    Could not locate executable ifort
    Could not locate executable ifl
    Could not locate executable f90
    Could not locate executable efl
    Could not locate executable gfortran
    Could not locate executable f95
    Could not locate executable g95
    Could not locate executable effort
    Could not locate executable efc
    don't know how to compile Fortran code on platform 'nt'
    non-existing path in 'numpy\\distutils': 'site.cfg'
    Running from numpy source directory.

    d:\local\temp\easy_install-dsrz9g\numpy-1.15.0rc2\setup.py:385: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
      run_build = parse_setuppy_commands()
    D:\python27\Lib\distutils\dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
      warnings.warn(msg)
    d:\local\temp\easy_install-dsrz9g\numpy-1.15.0rc2\numpy\distutils\system_info.py:625: UserWarning:
        Atlas (http://math-atlas.sourceforge.net/) libraries not found.
        Directories to search for the libraries can be specified in the
        numpy/distutils/site.cfg file (section [atlas]) or by setting
        the ATLAS environment variable.
      self.calc_info()

    d:\local\temp\easy_install-dsrz9g\numpy-1.15.0rc2\numpy\distutils\system_info.py:625: UserWarning:
        Blas (http://www.netlib.org/blas/) libraries not found.
        Directories to search for the libraries can be specified in the
        numpy/distutils/site.cfg file (section [blas]) or by setting
        the BLAS environment variable.
      self.calc_info()

    d:\local\temp\easy_install-dsrz9g\numpy-1.15.0rc2\numpy\distutils\system_info.py:625: UserWarning:
        Blas (http://www.netlib.org/blas/) sources not found.
        Directories to search for the sources can be specified in the
        numpy/distutils/site.cfg file (section [blas_src]) or by setting
        the BLAS_SRC environment variable.
      self.calc_info()

    d:\local\temp\easy_install-dsrz9g\numpy-1.15.0rc2\numpy\distutils\system_info.py:625: UserWarning:
        Lapack (http://www.netlib.org/lapack/) libraries not found.
        Directories to search for the libraries can be specified in the
        numpy/distutils/site.cfg file (section [lapack]) or by setting
        the LAPACK environment variable.
      self.calc_info()

    d:\local\temp\easy_install-dsrz9g\numpy-1.15.0rc2\numpy\distutils\system_info.py:625: UserWarning:
        Lapack (http://www.netlib.org/lapack/) sources not found.
        Directories to search for the sources can be specified in the
        numpy/distutils/site.cfg file (section [lapack_src]) or by setting
        the LAPACK_SRC environment variable.
      self.calc_info()

    D:\python27\Lib\distutils\dist.py:267: UserWarning: Unknown distribution option: 'define_macros'
      warnings.warn(msg)
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "D:\home\site\wwwroot\env\build\pandas\setup.py", line 743, in <module>
        **setuptools_kwargs)
      File "D:\python27\Lib\distutils\core.py", line 111, in setup
        _setup_distribution = dist = klass(attrs)
      File "D:\home\site\wwwroot\env\lib\site-packages\setuptools\dist.py", line 262, in __init__
        self.fetch_build_eggs(attrs['setup_requires'])
      File "D:\home\site\wwwroot\env\lib\site-packages\setuptools\dist.py", line 287, in fetch_build_eggs
        replace_conflicting=True,
      File "D:\home\site\wwwroot\env\lib\site-packages\pkg_resources.py", line 614, in resolve
        dist = best[req.key] = env.best_match(req, ws, installer)
      File "D:\home\site\wwwroot\env\lib\site-packages\pkg_resources.py", line 857, in best_match
        return self.obtain(req, installer)
      File "D:\home\site\wwwroot\env\lib\site-packages\pkg_resources.py", line 869, in obtain
        return installer(requirement)
      File "D:\home\site\wwwroot\env\lib\site-packages\setuptools\dist.py", line 338, in fetch_build_egg
        return cmd.easy_install(req)
      File "D:\home\site\wwwroot\env\lib\site-packages\setuptools\command\easy_install.py", line 613, in easy_install
        return self.install_item(spec, dist.location, tmpdir, deps)
      File "D:\home\site\wwwroot\env\lib\site-packages\setuptools\command\easy_install.py", line 643, in install_item
        dists = self.install_eggs(spec, download, tmpdir)
      File "D:\home\site\wwwroot\env\lib\site-packages\setuptools\command\easy_install.py", line 833, in install_eggs
        return self.build_and_install(setup_script, setup_base)
      File "D:\home\site\wwwroot\env\lib\site-packages\setuptools\command\easy_install.py", line 1055, in build_and_install
        self.run_setup(setup_script, setup_base, args)
      File "D:\home\site\wwwroot\env\lib\site-packages\setuptools\command\easy_install.py", line 1043, in run_setup
        raise DistutilsError("Setup script exited with %s" % (v.args[0],))
    distutils.errors.DistutilsError: Setup script exited with error: Microsoft Visual C++ 9.0 is required (Unable to find vcvarsall.bat). Get it from http://aka.ms/vcpython27
    Complete output from command python setup.py egg_info:

Could not locate executable g77
Could not locate executable f77
Could not locate executable ifort
Could not locate executable ifl
Could not locate executable f90
Could not locate executable efl
Could not locate executable gfortran
Could not locate executable f95
Could not locate executable g95
Could not locate executable effort
Could not locate executable efc

don't know how to compile Fortran code on platform 'nt'
non-existing path in 'numpy\\distutils': 'site.cfg'
Running from numpy source directory.
d:\local\temp\easy_install-dsrz9g\numpy-1.15.0rc2\setup.py:385: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
  run_build = parse_setuppy_commands()
D:\python27\Lib\distutils\dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
  warnings.warn(msg)

d:\local\temp\easy_install-dsrz9g\numpy-1.15.0rc2\numpy\distutils\system_info.py:625: UserWarning:
    Atlas (http://math-atlas.sourceforge.net/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [atlas]) or by setting
    the ATLAS environment variable.
  self.calc_info()

d:\local\temp\easy_install-dsrz9g\numpy-1.15.0rc2\numpy\distutils\system_info.py:625: UserWarning:
    Blas (http://www.netlib.org/blas/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [blas]) or by setting
    the BLAS environment variable.
  self.calc_info()

d:\local\temp\easy_install-dsrz9g\numpy-1.15.0rc2\numpy\distutils\system_info.py:625: UserWarning:
    Blas (http://www.netlib.org/blas/) sources not found.
    Directories to search for the sources can be specified in the
    numpy/distutils/site.cfg file (section [blas_src]) or by setting
    the BLAS_SRC environment variable.
  self.calc_info()

d:\local\temp\easy_install-dsrz9g\numpy-1.15.0rc2\numpy\distutils\system_info.py:625: UserWarning:
    Lapack (http://www.netlib.org/lapack/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [lapack]) or by setting
    the LAPACK environment variable.
  self.calc_info()

d:\local\temp\easy_install-dsrz9g\numpy-1.15.0rc2\numpy\distutils\system_info.py:625: UserWarning:
    Lapack (http://www.netlib.org/lapack/) sources not found.
    Directories to search for the sources can be specified in the
    numpy/distutils/site.cfg file (section [lapack_src]) or by setting
    the LAPACK_SRC environment variable.
  self.calc_info()

D:\python27\Lib\distutils\dist.py:267: UserWarning: Unknown distribution option: 'define_macros'
  warnings.warn(msg)

Traceback (most recent call last):
  File "<string>", line 17, in <module>
  File "D:\home\site\wwwroot\env\build\pandas\setup.py", line 743, in <module>
    **setuptools_kwargs)
  File "D:\python27\Lib\distutils\core.py", line 111, in setup
    _setup_distribution = dist = klass(attrs)
  File "D:\home\site\wwwroot\env\lib\site-packages\setuptools\dist.py", line 262, in __init__
    self.fetch_build_eggs(attrs['setup_requires'])
  File "D:\home\site\wwwroot\env\lib\site-packages\setuptools\dist.py", line 287, in fetch_build_eggs
    replace_conflicting=True,
  File "D:\home\site\wwwroot\env\lib\site-packages\pkg_resources.py", line 614, in resolve
    dist = best[req.key] = env.best_match(req, ws, installer)
  File "D:\home\site\wwwroot\env\lib\site-packages\pkg_resources.py", line 857, in best_match
    return self.obtain(req, installer)
  File "D:\home\site\wwwroot\env\lib\site-packages\pkg_resources.py", line 869, in obtain
    return installer(requirement)
  File "D:\home\site\wwwroot\env\lib\site-packages\setuptools\dist.py", line 338, in fetch_build_egg
    return cmd.easy_install(req)
  File "D:\home\site\wwwroot\env\lib\site-packages\setuptools\command\easy_install.py", line 613, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "D:\home\site\wwwroot\env\lib\site-packages\setuptools\command\easy_install.py", line 643, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "D:\home\site\wwwroot\env\lib\site-packages\setuptools\command\easy_install.py", line 833, in install_eggs
   return self.build_and_install(setup_script, setup_base)
  File "D:\home\site\wwwroot\env\lib\site-packages\setuptools\command\easy_install.py", line 1055, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "D:\home\site\wwwroot\env\lib\site-packages\setuptools\command\easy_install.py", line 1043, in run_setup
    raise DistutilsError("Setup script exited with %s" % (v.args[0],))

distutils.errors.DistutilsError: Setup script exited with error: Microsoft Visual C++ 9.0 is required (Unable to find vcvarsall.bat). Get it from http://aka.ms/vcpython27

----------------------------------------

Cleaning up...

Command python setup.py egg_info failed with error code 1 in D:\home\site\wwwroot\env\build\pandas
Storing debug log for failure in D:\home\pip\pip.log

An error has occurred during web site deployment.
\r\nD:\Program Files (x86)\SiteExtensions\Kudu\75.10629.3460\bin\Scripts\starter.cmd "D:\home\site\deployments\tools\deploy.cmd"

This guide is a reflection on how to use Wheel files to install Modules that cannot natively be installed via pip due to a compiler missing in the Azure App Service:

Microsoft Official documentation can be found here: https://docs.microsoft.com/en-us/azure/app-service/web-sites-python-configure#troubleshooting—package-installation

Tutorial

  1. Modify requirements.txt file
    1. Add the following item as the first line to the document:
      1. –find-links wheelhouse
        1. Note: If you do not have a requirements.txt file, you can simply create a new text document and add this line to it.  The requirements.txt file is what allows the Azure App Service to automatically go out and try and download packages you may need for your application.  Official documentation on this file is found here: https://docs.microsoft.com/en-us/azure/app-service/web-sites-python-configure#package-management
    2. Navigate to the Kudu Debug Console by going to https://yourappservice.scm.azurewebsites.net/DebugConsole
    3. Within the debug console, navigate to your version of Python.
      1. Note: The default Python versions in an Azure App Service are 2.7 and 3.4; however since Wheel will need to install some files, you cannot leverage the default directories of D:\Python27 for v2.7 and D:\Python34 for v3.4
      2. In this case, I’d recommend leveraging Extensions to install whatever version of Python.  Documentation on this can be found here: https://blogs.msdn.microsoft.com/pythonengineering/2016/08/04/upgrading-python-on-azure-app-service/
    4. Install the Python Wheel module:
      1. python.exe -m pip install wheel
    5. Obtain Wheel files
      1. Option 1: Build your own wheel files
        1. Execute the following command:
          1. python.exe -m pip wheel -r D:\home\site\wwwroot\requirements.txt -w wheelhouse

      2. Option 2: Obtain Wheel files
        1. Create a wheelhouse folder within your python directory
          1. mkdir wheelhouse

        2. Copy whl files to this directory
          1. You can obtain wheel files from PyPi or from Laboratory for Fluorescence Dynamics, University of California, Irvine.
            1. PyPi: Search for the module and then clicking on the Download Files button
              1. https://pypi.org/
            2. Laboratory for Fluorescence Dynamics, University of California, Irvine: Simply download the appropriate whl file listed on the page below
              1. https://www.lfd.uci.edu/~gohlke/pythonlibs/
    6. Install Modules
      1. Manual Install
        1. Execute the following command:
          python.exe -m pip install –upgrade -r D:\home\site\wwwroot\requirements.txt

      2. Deployment Install (from CI/CD pipeline)
        1. Configure .deployment and deploy.cmd file
          1. Official documentation on this can be found here: https://github.com/projectkudu/kudu/wiki/Custom-Deployment-Script
          2. .deployment file
            1. [config]
              command = deploy.cmd
          3. deploy.cmd file (modify the python directory to reflect your version)
            1. :: 1. Install Wheel
              echo Configure Wheel
              D:\home\python364x64\python.exe -m pip install wheel:: 2. Install packages
              echo Pip install requirements.
              D:\home\python364x64\python.exe -m pip install –upgrade -r D:\home\site\wwwroot\requirements.txt

At this point, the modules in question should be installed and ready for use! ūüôā

Setting up WeeWX with a Raspberry PI

This is a quick setup guide on how to configure the open source software WeeWX for a Personal Weather Station (PWS).  I highly recommend you check out the WeeWX User Guide as this information is very well documented.  Here is a reflection of how I was able to get WeeWX installed on a Raspberry PI with a brand new weather station.

  1. Setup your Raspberry PI
    1. How to setup your Raspberry PI: http://jackstromberg.com/2018/03/setting-up-a-new-raspberry-pi-via-ssh/
      1. Note: Raspbian is a distribution based upon Debian.  In this case, we will follow the Debian instructions for setting up WeeWX.
        1. http://weewx.com/docs/debian.htm
  2. (Optional) Configure the Raspberry PI to be localized to your environment
    1. sudo raspi-config
      1. Here you can arrow down to Localization Options and configure the timezone to match that of your console/weather sensor.  Keeping time is critical, so if possible, try to keep the date/time between your weather station and the Raspberry PI as close as possible.
  3. Configure Apt-Get to look for the WeeWX packages
    wget -qO - http://weewx.com/keys.html | sudo apt-key add -
    sudo wget -qO - http://weewx.com/apt/weewx.list | sudo tee /etc/apt/sources.list.d/weewx.list
  4. Update your Raspberry-PI to use the latest packages
    sudo apt-get update
    sudo apt-get upgrade
  5. Before installation, ensure you have your console or device setup and connected to your Raspberry PI for WeeWX to pull the data
  6. Determine the interface the console is connected to (if using a directly attached data loggerm skip if using an IP based source)
    1. Execute the command dmesg and look for what interface the data logger is connected to
      1. In my example, you can see the data logger is connected to ttyUSB0
  7. Launch the installation wizard for weewx
    1. sudo apt-get install weewx
      1. Note: You will likely be prompted to install a few dependencies, type Y for yes to install them
  8. Installation
    1. Enter the location of your weather station: Santa’s Workshop, North Pole
    2. Enter in the latitude, longitude of your weather station
      1. Note: If you don’t have GPS, you can easily find this by using Bing Maps or Google Maps, navigating to your location, and right clicking.
        1. For Bing, it will just show you the lat/long values when you right click
        2. For Google, click on “What’s Here” and it will list these values
      2. Note: You can be more specific than 3 digits behind the decimal, so if you want to use a more specific set of coordinates like 40.689167, -74.044444, that is acceptable.
    3. Enter in your Altitude of where the weather station is
      1. You can use Google Earth to find the altitude or this tool here: https://www.freemaptools.com/elevation-finder.htm
    4. Set your preferred unit of measurement
      1. US (Imperial) or Metric
    5. Select your weather station type
      1. I.e. AcuRite, Vantage (if using Davis), etc.
    6. Select the interface the device is listening on
    7. For those using serial port, select the interface that the data logger is connected to.  You should have found this in step 4 above; if using ethernet, go ahead and type in the IP, Port, etc. of the data logger.
  9. At this point WeeWX is technically installed, however many individuals will want to present the WeeWX reports via webpage.  In this case, we’ll install nginx, which is a lightweight webserver
    1. sudo apt-get install nginx
      1. More details on this can be found here: http://www.weewx.com/docs/usersguide.htm#integrating_with_webserver
  10. Configure WeeWX to minimize disk IO
    1. Why do we need to do this?  Since Raspberry PI’s leverage SD cards, there is typically a finite number of reads/writes to the SD Card.  In this case, it is recommended to either leverage an external database/fileserver for WeeWX to write its reports.  Alternatively, we can also configure WeeWX to leverage ram to host the reports, which will prevent IO to the SD card (in this case, theoretically increasing the life of the drive)
      1. Three approaches are outlined here–in this guide I’ll reflect the GitHub page in saving reports to a temporary file system using tmpfs
        1. Add an entry to fstab
          1. echo “weewx_reports /var/weewx/reports tmpfs size=20M,noexec,nosuid,nodev 0 0” | sudo tee -a /etc/fstab
        2. Mount the new file system
          1. sudo mkdir -p /var/weewx/reports
          2. sudo mount -a
        3. Update weewx.config file to point to new directory
          1. sudo sed -i -e ‘s%HTML_ROOT =.*%HTML_ROOT = /var/weewx/reports%’ /etc/weewx/weewx.conf
        4. Restart WeeWX service
          1. sudo service weewx restart
        5. Create symbolic link to point webserver to the reports
          1. sudo ln -s /var/weewx/reports /var/www/html/weewx
        6. Give the web server the ability to read from the directory
          1. sudo chmod -R 755 /var/www/html/weewx

At this point, go ahead and browse out to http://youripaddress/weewx/ to see your weather.

Notes:

WeeWX updates the webpage every 30 minutes (1800 seconds) out of the box.  You can force a report update by executing wee_reports weewx.conf or you can modify the /etc/weewx/weewx.conf file by changing the archive_interval variable (in seconds) under the [StdArchive] section.

You can modify the Weewx configuration by editing: /etc/weewx/weewx.conf

You can validate if WeeWX is running by executing: service weewx status

You can look at diagnostics logs by following the guide here: http://www.weewx.com/docs/usersguide.htm#monitoring

Best practices guide on using WeeMX + Raspberry PI: https://github.com/weewx/weewx/wiki/Raspberry%20Pi

How to upgrade your Windows Server Evaluation/Trial

Scenario: You downloaded the evaluation copy of Windows Server and you have 180 days to test out whatever you are working on.¬† Fast forward a few months and you only have a few days left and you are so happy with how it works, you go out and buy the whole license key.¬† When you go to apply the license key under System, you get a big ol’ error that says: “This edition cannot be upgraded.”

Solution:

You can use the DISM tool to figure out what versions of Windows Server you can upgrade to, and also use the tool to help change the product key of the version installed.

Easy enough, let’s go ahead and open up command prompt as an administrator (right click on windows flag/start icon, Command Prompt (Admin):

Execute the following command to find out what versions you can upgrade to:

Dism /Online /Get-TargetEditions

In this case, you can see I can upgrade to ServerStandard or ServerDatacenter

Next, let’s go ahead and actually upgrade the edition and inject my license key:

Dism /Online /Set-Edition:TheEditionListedYouWantToGoTo /AcceptEula /ProductKey:XXXXX-XXXXX-XXXXX-XXXXX-XXXXX

Viola!  At this point, I simply need to reboot and my instance will be upgraded accordingly.

[Tutorial] Integrate Visual Studio Code with Visual Studio Team Services

Here’s a quick way to integrate Visual Studio Code with Visual Studio Team Services.

  1. Create a new Team Project
    1. Instructions on how to create a new Team Project are outlined here: https://docs.microsoft.com/en-us/vsts/accounts/create-account-msa-or-work-student
  2. Create a Personal Access Token
    1. Instructions on how to generate a personal access token are outlined here: https://docs.microsoft.com/en-us/vsts/accounts/use-personal-access-tokens-to-authenticate
  3. Download and install Git: https://git-scm.com/download/
  4. Download Visual Studio Code: https://code.visualstudio.com/Download
  5. Inside of Visual Studio Code, click on the Extensions button
  6. Search the marketplace for Visual Studio Team Services and select Install button
  7. Once the extension has been Installed, click on the Reload button.
  8. Inside of Visual Studio Code, press F1 on your keyboard and type Git: Clone

  9. Once prompted, type in the URL to your Team Project and click on the Open Repository button
  10. Once in the repository, type: Team: Sign In  Select Provide an Access token manually, enter the Personal Access Token from Visual Studio Online and press Enter on your keyboard
  11. From there, go ahead and make a change to any of the files in your Repository
  12. Click on the Source Control icon in Visual Studio Code
  13. Select Commit All
    1. Note: You will be prompted to type in a commit message, go ahead and type in what you changed

  14. Either select Push from the button in the top right, or click the Push button in the bottom left corner
  15. Validate you see the committed changes in Visual Studio Team Services

Windows 10 – Missing Windows Disc Image Burner for ISO files

In Windows, you typically are able to download a .ISO file, right click on it, and burn it via your CD/DVD drive using the Windows Disc Image Burner application. Unfortunately, for whatever reason my machine is missing this menu item.

A quick workaround that doesn’t involve any registry hacks is to simply right click on the file, select Open With, and select Choose another app.

Select More apps and scroll to the bottom and select Look for another app on this PC.

Navigate to C:\Windows\System32, select isoburn.exe, and click Open

At this point, you can go ahead and burn your iso ūüôā

 

[Tutorial] Gathering trace/event logs in ADFS v2.0 and v3.0

Problem:

Gathering trace/event logs in ADFS is not a trivial task.  The following article will show you how to gather these logs to further help investigate relying party trust issues or issues with end users authenticating to the service.  This tutorial will be leveraging ADFS v3.0 on Server 2012 R2.  The same steps should apply for v2.0 on Server 2008 R2.  This process does change slighting in ADFS on Server 2016 as the logging engine was rewritten.  Depending on demand, a second article will be released for ADFS on Server 2016.

Caviets:

Before beginning, as a side note, debugging in ADFS v2-3¬†is honestly a total PITA (pain in the… butt).¬† The problem with ADFS logging is logs are stored on the machines serving the requests, not centrally.¬† In this case, you will likely have to enable tracing on each ADFS server, or configure your load balancer/host file¬†to temporarly route requests to a specific machine so you know which server to hunt down for the logs.¬† Likewise, as you will find at the end of the tutorial, the logs gathered from ADFS are very verbose.¬† Take some time to familiarize yourself with the logs of a working request vs a failure to get used to what logs are actually meaningful.

Tutorial:

Enable list of events/audits to be logged

  1. Login to one of your ADFS servers that you believe will be authenticating the end users
  2. Open Server Manager

  3. In Server Manager, select Tools -> AD FS Management
  4. In AD FS Management, select AD FS in the top left and select Edit Federation Service Properties…
  5. Click on the Events tab and check all the items you wish to log and click OK

Enable tracing

  1. Open Server Manager
  2. Select Tools -> Event Viewer
  3. In Event Viewer, select View in the top menu, and select Show Analytic and Debug Logs
  4. Expand Applications and Services Logs, expand AD FS Tracing, and select Debug
    1. Note: In ADFS v2, the AD FS Tracing folder will be called AD FS 2.0 Tracing
  5. When you are ready to begin collecting logs, right click on Debug and select Enable Log
  6. Click OK when prompted to write over the existing event logs
    1. Note: Each time you enable/disable AD FS Tracing, Event Viewer will purge your last results.  I highly recommend you export your logs if you need them for comparison at a later time.
  7. At this point, recreate the issue, error, or login to the relying party you want to debug.
  8. Once you have recreated the error or logged in, go back to Event Viewer, right click on Debug and select Disable Log
  9. At this point, you should have some events captured to further analyse ūüôā
  10. Optional Step: Right click on Debug and select Save All Events As…¬† This will export to a evtx file, in which this can be sent to another team for analysis or you can reference the logs at a later time.
    1. Note: If you are sending the events over to another team for analysis, zip the logs as it will greatly decrease the file size ūüôā

Common error when enabling Debug logging

One error I typically see is the following:

AD FS Debug РThe requested operation cannot be performed over an enabled direct channel.  The channel must first be disabled before performing the requested operation

This error is caused by a misconfiguration on the logging properties of the Debug log.  Please verify that you have not manually enabled the debug log nor have the maximum log file size set to Overwrite events as needed.

To fix, right click on Debug and select Properties

Typically, the screenshot below is an example of the incorrect settings used; make sure that Enable Logging is unchecked and is Do not overwrite events ( Clear logs manually ) is checked

Here is a picture of the correct settings for the AD FS Tracing Debug Logs; at which point, once the settings are applied, you should no longer receive this error when conducting your debug/trace logging.

[Tutorial] Using Fiddler to debug SAML tokens on Mobile Devices (Android)

Use Case:

This guide will go over configuring Fiddler to intercept traffic from mobile devices for debugging purposes.  This scenario can be beneficial in tracing/debugging SAML tokens issued from your IdP for a mobile application to consume.  We will be able to validate all traffic flowing in/from the Android device.

Configuring/Setting up Fiddler:

  1. Grab the latest copy of Fiddler from their website for Windows (it is a free download)
    1. https://www.telerik.com/download/fiddlerDownload Fiddler
  2. Install Fiddler on your local machine
    1. Double click fiddlersetup.exe
      Run fiddlersetup
    2. Agree to the End User License Agreement
      Fiddler Install - Accept EULA
    3. Set the installation directory and click Install
      Fiddler Install - Destination Folder
    4. Close the setup wizard
      Fiddler Install - Close Installation
  3. Launch Fiddler
    Launch Fiddler - Windows 10
  4. Click Cancel if prompted about AppContainers
    Fiddler - AppContainer Configuration - Cancel
  5. With Fiddler open click on Tools -> Telerik Fiddler Options…
    Fiddler - Tools - Telerik Fiddler Options
  6. Click on the Connections tab and check Allow remote computers to connect

  7. You will receive a dialog box saying it will need to restart.  Click OK and close out of Fiddler
  8. Once you relaunch Fiddler, click on the down arrow (if shown) and hover over the Online icon

At this point, Fiddler is configured properly, let’s shift over to your mobile device. We’ll shift gears to configuring the Android device to push traffic to Fiddler.

Configuring an Android device
(Android v6.0.1 at the time of writing)

  1. Slide down the notifications drawer from the top of the screen and hit the Settings (gear) icon in the top right
  2. Select Wi-Fi under the Wireless and networks section
  3. Select the wireless network you are connected to and click Edit

  4. Scroll down and check Show advanced options

  5. Select the drop-down for Proxy and choose Manual

  6. Type in the IP address gathered from Fiddler for the Proxy host name and set the Proxy Port to 8888 and click Save
    1. Note: 8888 is the default port for Fiddler, the port can be found under Fiddler -> Telerik Fiddler Options -> Connections tab
  7. Next, open up your web browser and navigate to http://ipv4.fiddler:8888
    1. Note: This is a small webpage served by the Fiddler application to validate the proxy settings are correct.  Likewise, we will use this page in the next step for SSL decryption
  8. On the Fiddler Echo Service page, click on the You can download the FiddlerRoot Certificate link
    1. Note: This download Fiddler’s root certificate to allow us to intercept SSL traffic for debugging purposes
  9. Once the certificate has downloaded, type Fiddler as the Certificate name and click OK

  10. Optional step: Open up your web browser and navigate to a website using SSL (I did https://google.com)
    1. Note: Here you can validate that the SSL certificate used is Fiddler’s root certificate.¬† This is a good sign that we are intercepting the traffic

Turn off Fiddler from intercepting SSL traffic

Remove the proxy settings

  1. Slide down the notifications drawer from the top of the screen and hit the Settings (gear) icon in the top right
  2. Select Wi-Fi under the Wireless and networks section
  3. Select the wireless network you are connected to and click Edit

  4. Scroll down and check Show advanced options (you should see your old proxy settings unlike my screenshot below)

  5. Select the drop-down for Proxy and choose None

  6. Select Save
  7. At this point, you should be able to capture the traffic through the Fiddler application on your Windows machine; see the screenshot below showing traffic from the android device
    1. NOTE/TIP: If you turn off capturing, you will turn off capturing on Windows, but not for the mobile device.¬† This can help cut down on the “noise” in getting your sample/debug logs.

Remove the Fiddler SSL certificate

  1. Slide down the notifications drawer from the top of the screen and hit the Settings (gear) icon in the top right
  2. Select Security

  3. Select Trusted credentials

  4. Select the User tab on the Trusted credentials window
  5. Scroll down through the certificate information and towards the bottom you will see a Remove button; press the REMOVE button.
    1. Note: You have to scroll the text, there is no scrollbar until you start the scrolling gesture

 

List of time zones consumed by Azure

When creating Azure Automation scripts, you may have to reference time zones by name.  Below is a table of acceptable values you may use in your scripts to denote the proper time zone.

Name of Time Zone Time
Dateline Standard Time (UTC-12:00) International Date Line West
UTC-11 (UTC-11:00) Coordinated Universal Time-11
Hawaiian Standard Time (UTC-10:00) Hawaii
Alaskan Standard Time (UTC-09:00) Alaska
Pacific Standard Time (Mexico) (UTC-08:00) Baja California
Pacific Standard Time (UTC-08:00) Pacific Time (US & Canada)
US Mountain Standard Time (UTC-07:00) Arizona
Mountain Standard Time (Mexico) (UTC-07:00) Chihuahua, La Paz, Mazatlan
Mountain Standard Time (UTC-07:00) Mountain Time (US & Canada)
Central America Standard Time (UTC-06:00) Central America
Central Standard Time (UTC-06:00) Central Time (US & Canada)
Central Standard Time (Mexico) (UTC-06:00) Guadalajara, Mexico City, Monterrey
Canada Central Standard Time (UTC-06:00) Saskatchewan
SA Pacific Standard Time (UTC-05:00) Bogota, Lima, Quito, Rio Branco
Eastern Standard Time (Mexico) (UTC-05:00) Chetumal
Eastern Standard Time (UTC-05:00) Eastern Time (US & Canada)
US Eastern Standard Time (UTC-05:00) Indiana (East)
Venezuela Standard Time (UTC-04:30) Caracas
Paraguay Standard Time (UTC-04:00) Asuncion
Atlantic Standard Time (UTC-04:00) Atlantic Time (Canada)
Central Brazilian Standard Time (UTC-04:00) Cuiaba
SA Western Standard Time (UTC-04:00) Georgetown, La Paz, Manaus, San Juan
Newfoundland Standard Time (UTC-03:30) Newfoundland
E. South America Standard Time (UTC-03:00) Brasilia
SA Eastern Standard Time (UTC-03:00) Cayenne, Fortaleza
Argentina Standard Time (UTC-03:00) City of Buenos Aires
Greenland Standard Time (UTC-03:00) Greenland
Montevideo Standard Time (UTC-03:00) Montevideo
Bahia Standard Time (UTC-03:00) Salvador
Pacific SA Standard Time (UTC-03:00) Santiago
UTC-02 (UTC-02:00) Coordinated Universal Time-02
Azores Standard Time (UTC-01:00) Azores
Cape Verde Standard Time (UTC-01:00) Cabo Verde Is.
Morocco Standard Time (UTC) Casablanca
UTC (UTC) Coordinated Universal Time
GMT Standard Time (UTC) Dublin, Edinburgh, Lisbon, London
Greenwich Standard Time (UTC) Monrovia, Reykjavik
W. Europe Standard Time (UTC+01:00) Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna
Central Europe Standard Time (UTC+01:00) Belgrade, Bratislava, Budapest, Ljubljana, Prague
Romance Standard Time (UTC+01:00) Brussels, Copenhagen, Madrid, Paris
Central European Standard Time (UTC+01:00) Sarajevo, Skopje, Warsaw, Zagreb
W. Central Africa Standard Time (UTC+01:00) West Central Africa
Namibia Standard Time (UTC+01:00) Windhoek
Jordan Standard Time (UTC+02:00) Amman
GTB Standard Time (UTC+02:00) Athens, Bucharest
Middle East Standard Time (UTC+02:00) Beirut
Egypt Standard Time (UTC+02:00) Cairo
Syria Standard Time (UTC+02:00) Damascus
E. Europe Standard Time (UTC+02:00) E. Europe
South Africa Standard Time (UTC+02:00) Harare, Pretoria
FLE Standard Time (UTC+02:00) Helsinki, Kyiv, Riga, Sofia, Tallinn, Vilnius
Turkey Standard Time (UTC+02:00) Istanbul
Israel Standard Time (UTC+02:00) Jerusalem
Kaliningrad Standard Time (UTC+02:00) Kaliningrad (RTZ 1)
Libya Standard Time (UTC+02:00) Tripoli
Arabic Standard Time (UTC+03:00) Baghdad
Arab Standard Time (UTC+03:00) Kuwait, Riyadh
Belarus Standard Time (UTC+03:00) Minsk
Russian Standard Time (UTC+03:00) Moscow, St. Petersburg, Volgograd (RTZ 2)
E. Africa Standard Time (UTC+03:00) Nairobi
Iran Standard Time (UTC+03:30) Tehran
Arabian Standard Time (UTC+04:00) Abu Dhabi, Muscat
Azerbaijan Standard Time (UTC+04:00) Baku
Russia Time Zone 3 (UTC+04:00) Izhevsk, Samara (RTZ 3)
Mauritius Standard Time (UTC+04:00) Port Louis
Georgian Standard Time (UTC+04:00) Tbilisi
Caucasus Standard Time (UTC+04:00) Yerevan
Afghanistan Standard Time (UTC+04:30) Kabul
West Asia Standard Time (UTC+05:00) Ashgabat, Tashkent
Ekaterinburg Standard Time (UTC+05:00) Ekaterinburg (RTZ 4)
Pakistan Standard Time (UTC+05:00) Islamabad, Karachi
India Standard Time (UTC+05:30) Chennai, Kolkata, Mumbai, New Delhi
Sri Lanka Standard Time (UTC+05:30) Sri Jayawardenepura
Nepal Standard Time (UTC+05:45) Kathmandu
Central Asia Standard Time (UTC+06:00) Astana
Bangladesh Standard Time (UTC+06:00) Dhaka
N. Central Asia Standard Time (UTC+06:00) Novosibirsk (RTZ 5)
Myanmar Standard Time (UTC+06:30) Yangon (Rangoon)
SE Asia Standard Time (UTC+07:00) Bangkok, Hanoi, Jakarta
North Asia Standard Time (UTC+07:00) Krasnoyarsk (RTZ 6)
China Standard Time (UTC+08:00) Beijing, Chongqing, Hong Kong, Urumqi
North Asia East Standard Time (UTC+08:00) Irkutsk (RTZ 7)
Singapore Standard Time (UTC+08:00) Kuala Lumpur, Singapore
W. Australia Standard Time (UTC+08:00) Perth
Taipei Standard Time (UTC+08:00) Taipei
Ulaanbaatar Standard Time (UTC+08:00) Ulaanbaatar
Tokyo Standard Time (UTC+09:00) Osaka, Sapporo, Tokyo
Korea Standard Time (UTC+09:00) Seoul
Yakutsk Standard Time (UTC+09:00) Yakutsk (RTZ 8)
Cen. Australia Standard Time (UTC+09:30) Adelaide
AUS Central Standard Time (UTC+09:30) Darwin
E. Australia Standard Time (UTC+10:00) Brisbane
AUS Eastern Standard Time (UTC+10:00) Canberra, Melbourne, Sydney
West Pacific Standard Time (UTC+10:00) Guam, Port Moresby
Tasmania Standard Time (UTC+10:00) Hobart
Magadan Standard Time (UTC+10:00) Magadan
Vladivostok Standard Time (UTC+10:00) Vladivostok, Magadan (RTZ 9)
Russia Time Zone 10 (UTC+11:00) Chokurdakh (RTZ 10)
Central Pacific Standard Time (UTC+11:00) Solomon Is., New Caledonia
Russia Time Zone 11 (UTC+12:00) Anadyr, Petropavlovsk-Kamchatsky (RTZ 11)
New Zealand Standard Time (UTC+12:00) Auckland, Wellington
UTC+12 (UTC+12:00) Coordinated Universal Time+12
Fiji Standard Time (UTC+12:00) Fiji
Tonga Standard Time (UTC+13:00) Nuku’alofa
Samoa Standard Time (UTC+13:00) Samoa
Line Islands Standard Time (UTC+14:00) Kiritimati Island

[Tutorial] Using Fiddler to debug SAML tokens issued from ADFS

Problem:

Many applications want to federate with leverage certain attributes like nameid (nameidentifier), but the problem is the format is wildly different from one application to another.¬† In this case, one application might use a unique value like an employee ID, another UPN, another email address, and so on.¬† Or maybe it isn’t an attribute, but you are leveraging SHA1 as your signature hashing algorithm and the application is looking for MD5.

In this case, sometimes you may not be sure what you are sending to the application and are looking to the vendor to help you understand what you need to change in ADFS or if you are working on a custom application, need help debugging your claims rules to integrate into that application.  In this case, I will show you how to leverage Fiddler to acquire the SAML Tokens issued by ADFS to validate what attributes/values you are passing to the federate application.

Tutorial:

  1. Grab the latest copy of Fiddler from their website (it is a free download)
    1. https://www.telerik.com/download/fiddlerDownload Fiddler
  2. Install Fiddler on your local machine
    1. Double click fiddlersetup.exe
      Run fiddlersetup
    2. Agree to the End User License Agreement
      Fiddler Install - Accept EULA
    3. Set the installation directory and click Install
      Fiddler Install - Destination Folder
    4. Close the setup wizard
      Fiddler Install - Close Installation
  3. Launch Fiddler
    Launch Fiddler - Windows 10
  4. Click Cancel if prompted about AppContainers
    Fiddler - AppContainer Configuration - Cancel
  5. With Fiddler open click on Tools -> Telerik Fiddler Options…
    Fiddler - Tools - Telerik Fiddler Options
  6. Click on the HTTPS tab and check Decrypt HTTPS traffic and click OK
    1. Note: you may be prompted to trust a certificate.  You must trust the certificate so Fiddler can intercept your encrypted traffic and decrypt it.  Fiddler will not permanently capture traffic when the application is closed.
      Fiddler - Tools - Telerik Fiddler Options - HTTPS - Decrypt HTTPS traffic
  7. Close out of Fiddler
    Fiddler - Close
  8. Open Fiddler
    Launch Fiddler - Windows 10
  9. Open up Internet Explorer in one window and Fiddler side-by-side.  Drag the Crosshair icon onto Internet Explorer.  This will target only traffic in this process (browser window) to help filter down intercepted traffic.
    Fiddler - Process Selector - Drag Drop
  10. Select the X icon with a dropdown and click Remove all to clear your trace
    Fiddler - X - Remove All
  11. Go to the url of the federated application and login.  In this case, I am going to use https://outlook.com/owa/jackstromberg.com; once you have logged into the application or received the error to your application upon login, click FileCapture Traffic to stop the logs
    Fiddler - File - Capture Traffic - ADFS
  12. Within your logs, look for the last 200 response from your ADFS server before being redirected to your application (which will not show up as a 302, since we are posting to the new URL)
    Fiddler - HTTPS 200 - ADFS - SAML Post
  13. Click on the Inspectors tab, and select the Raw tab at the bottom and copy the value from the hidden input tag with the name of wresult
    Fiddler - Inspectors - Raw - wresult - encoded html
  14. Paste the encoded HTML into my HTML Encoder/Decoder in the Encoded text box and click Decode.
    1. Note: The encoder/decoder is all JavaScript based that functions client/side, so no data will leave your network.
      JackStromberg - HTML Encoder - Decoder - SAML
  15. Copy the Decoded HTML and paste it into an XML formatter of your choice.  Here I am using Bing:
    Bing - XML Formatter - SAML Token
  16. Copy the result into Notepad and you can now read the information
    Notepad - SAML Decoded - Formatted XML

Going into the claim and how it works is outside the scope of this tutorial, but as you can see in the last screenshot above we have the raw SAML token we will send to the relying party trust to consume.  At this point, the vendor can be involved to help troubleshoot any values or attributes that are in an incorrect format.

Creating self-signed certificates with makecert

If you are even in a bind and need a quick self-signed SSL cert and have the Windows SDK installed on your machine, there’s a chance you may have the makecert utility and can generate a quick self-signed SSL cert.

The command is as follows:

makecert -r -pe -n “CN=SelfSigned SSL” -a sha1 -ss My -len 2048 -sy 24 -b 01/01/2015 -e 01/01/2050

A complete list of each of the switches can be found here:
https://msdn.microsoft.com/library/windows/desktop/aa386968.aspx

Additionally, a nice MSDN article has been posted on this subject as well: https://msdn.microsoft.com/en-us/library/ff699202.aspx

MakeCert is available as part of the Windows SDK, which you can download from http://go.microsoft.com/fwlink/p/?linkid=84091