Roman Kubica

Blog

  • Password page takes long time to load

    I got assigned task with two pages loading too long time. The first one was about custom form with password reset functionality (solved). The second one was poorly described and was about some certifications take long time to load (moved to next post).

    Password reset page

    Lets start with the custom page for password reset. Some basic info. Different roles have access to different users and acounts, e.g. manager, user administrator or basic user with admin accounts.

    Workflow did not contain extensive QueryOptions or context search, but the Form did have quite large QueryOption and quite a few filters for AND and OR clauses.

    My first approach and funny silly debug mistake.

    This was initital process to confirm that the slow performance is caused by the custom code and not by OOTB IIQ.

    This line is for future retrospective if ever changes: so far I have not found a better way on how to debug XML objects in IIQ. The only way is through “console output messages” into a log file.

    These are the last lines of the first script section.

    qo.addFilters(filters);
return (context.countObjects(qo,));

    I add lines above to the Form objects scrpt fields, like this

    log.error("script identity start")
...
qo.addFilters(filters);
log.error("script identity end");
return (context.countObjects(Links.class,qo) > 0);

    Second script for hidden variable

    log.error("script hidden start")
...
qo.addFilters(filters);
log.error("script hidden end");
return (context.countObjects(Links.class,qo) > 0);

    Now I open the password reset page and I get a bit confused about the output and timings between the steps in the log file.

    It shows some some 2 seconds between steps, rather than start/end log messages (not exact log messages)

    script identity end – 14:03:14
    script hidden start – 14:03:16

    Bit puzzled, but after a while i can see it clearly and smile about how silly i was 🙂

    Correcting the debug messages and confirming the delay is due to the countObjects DB search in both scripts. 

    log.error("script hidden start")
...
qo.addFilters(filters);
log.error("script hidden end1");
int objs = context.countObjects(Links.class,qo);
log.error("script hidden end2");
return (objs  > 0);

    Moving down from TEST to my DEV environment

    My spadmin in my own dev is set up without additional roles or anything, therefore the admin user cannot reset any passwords with the custom functionality. It is a good thing, because the query takes around 26 seconds and returns 0 records.

    Assumptions:
    It must go through whole table to find nothing. (correct)
    It must be due to missing index. (wrong)
    It should be easy to locate the DB query with trace enabled (correct)

    I used Dev tools in Edge and check the timings for the API call. I enabled trace on all objects in log4j2 and soon I was able to see the two queries.

    Combined select query with one inner and 3 left joins on Identity table. I am not database expert, but I could see that each left join is increased exponentially from the live statistics in MSSQL studio.

    I removed the left joins and related OR clauses. One inner and one left join as a result. Suddenly It returned results (still 0) within a second. I was on the right path.

    Reviewed the code for QueryOptions and filters. Logically it could have been split, so I did split it with some extra optimizing and added comments for my colleagues on why it is split in simialr query options and context.countObjects calls.

    Now it runs on my DEV within a second. Retested and confirmed usecases for different user types, deployed to TEST. 4 seconds to under 1 second(success). Finally created a PR and waiting for the next release to save 8 seconds in Production for each password reset.

    How did I debug in this case?

    • used error debug messages to find/confirm the problematic code
    • set up DEV environment to not run any tasks, that allows me to run trace on all objects and classes used by IIQ when needed
    • read trace logs for SQL queries and timings



  • IdentityIQ upgrade 8.4p2 and unrelated performance issues

    Written as a story, if you want to skip to Performance issue section, then scroll down to the end section to Solution.

    I have been working for 5 months on IdentityIQ upgrade project as technical lead. It went ok and it was delivered on time. Could not do it with the team help and coming together for regression testing and finalizing tasks.

    The upgrade weekend went throught with some minor issues. It was long. Minor mistakes too. I blame the rushed two last weeks with many tasks and many people involved. Either way overall success, very happy and tired.

    Next day minor performance issue, probably system catching up (tasks) after being off for 2 days. Tuesday, not so minor anymore. I was tryign to get involved, but was refused. In my opinion good decision on one hand as it might have been something else than upgrade and had to be confirmed before me joining. I was eventually called in for help week after to take over from my colleague, who did not deny or confirm upgrade as a cause.

    Performance issue

    The issue was about overall slowness affecting everything and everyone. It was not isolated part of the application like specific task or GUI section. It was everything, UI, Batch and all operations with spikes.

    What we knew that has happened, chronologically:
    IIQ upgrade (weekend)
    Minor performance issues (Monday)
    Major performance issues (Wednesday)
    Collapse (Friday)
    SQL server Patch (weekend)

    We have not upgraded Java and Tomcat, so yes, you guessing right it was either upgraded IIQ OOTB functionality, our custom code or Database. Right?

    For the first day, we were kind of jumping between IIQ and DB and trying to debug different things. Bit of blaming and also clarifying reasons for causing this or more specifically excluding reasons what it could not have been.

    Our first conclusion with DB team was that we need to rebuild indexes as it somehow helped previously with a lot of table updates (not much mine idea). It helped a bit for a day. Then we were hit again.

    I think at that point we were able to narrow it down to database, because the queries were spiking from IIQ servers, but also directly on the DB. Therefore it could have not been IIQ servers that cause the performance hit. That was a good conlusion.

    The only trouble was there was maybe 30-50% CPU utlization. RAM was used 90% and managed by SQL server to utilize. There were no visible spikes. That went for another day with some random queries testing and also whether the Perform maintenance tasks are the cause and slowing down everything else.

    In the next day I had some enlightment that it could be the Disk IO utilization. We haven’t received any info about it yet till now. Once provided , we could see it. Yes the utilization was 100% almost all the time. The spikes were visible in the graph and confirmed our behaviour. Finally moving somewhere – my first win.

    We were still trying to figure it out what is causing this. To be honest I didn’t know as I am not database expert, but for sure I knew there is something wrong with the Disk IO. Being tired of taking blame that it is due to upgrade. I requested a Disk IO statistics for 1 month to compare how it really as before the IIQ upgrade.

    Solution

    And there it was. I was angry, I was happy, I was cursing, I wanted to share it and kick some ass.

    The Disk IO statistics report as a image were ok, but there was a pattern of Disk IO upper limit as there was a horizontal line on a graph with 100% utilization since upgrade date

    At the first glance, it looked like since upgrade, but when I zoomed in and inspected the blurry image, it turned out it was limited a day or two before upgrade. – Yes, another win

    Raised the question with the DB/storage team and ofcourse there was a change for setting an upper limit for our service. marked as low impact, low risk change 🙂 I couldn’t believe it. The guy was very unproffesional and still blaming us for the change, but at the end reverted it (good push and good communication history proof from my colleague). I did not have that. I was pushing for change to be reverted due to incident.

    Once reverted and upper limit lifted. All went back to normal and we could finally leave for a weekend.

    Good approach

    Push the other teams or people to provide proof or gather more information

    Go for your hunch, but verify with proof or confirm logically

    Be the one to take a lead or stand up when needed

    Do single change at a time to find the rootcause

    Lessons learned

    Check for changes sooner when large incidents occur (coudl have been found sooner)

    Take a moment and brainstorm few theories in the beginning (this could increase finding the right path from the beginning)

    Avoid pushing your opinion without any proof (hunch is ok, but be reasonable)

  • Powershell overview and basics

    Run with Powershell
powershell.exe -command "& 'D:\powershell.ps1'" -ExecutionPolicy Bypass

    How long script takes?

    [datetime]$startDate = Get-Date
[datetime]$endDate = Get-Date
Write-Host $(NEW-TIMESPAN –Start $startDate –End $endDate )

    Find duplicate user values

    Get-ADUser -Filter { employeeID -like "*" } -SearchBase "OU=yourOU,DC=domain,DC=com" -property employeeID | Group-Object employeeID | Where-Object {$_.Count -ge 2} | select -ExpandProperty group | Select-Object Name, UserPrincipalName, SamAccountName, employeeID

  • Python

    Installation

    sudo apt install python3.10
sudo apt install python3-pip
pip install --user pipenv
python -m site --user-base
export PATH="$PATH:/pathAbove"
pipenv install
pipenv install flask

    Data structures

    Tuple

    Cannot edit after it is created – Immutable.

    thistuple = (“1”, “2”, “3”)
    for x in thistuple:
      print(x)

    List

    thislist = [“1”, “2”, “3”]
    for x in thislist:
      print(x)

  • Databases

    Postgres

    sudo apt install postgresql
    sudo service postgresql status
    sudo passwd postgres
    sudo -u postgres psql

    create database dbname;
create user username with encrypted password 'password';
grant all privileges on database dbname to username;

    Useful commands

    ALTER TABLE table ADD column dataType NOT NULL;

    \l list databases
    \c select database
    \dt list tables

  • Move a wordpress site to the new server

    Backup

    mysqldump -u root -p database1 > database.sql

    tar -zcvf data.tar.gz data

    Restore

    CREATE DATABASE database1;
    CREATE USER ‘user’@’localhost’ IDENTIFIED BY ‘pwd’;
    GRANT ALL PRIVILEGES ON database1 . * TO ‘user’@’localhost’;
    FLUSH PRIVILEGES;

    mysql -u root -p database1 < database.sql

    tar -zxvf data.tar.gz
    new website: 

    curl https://wordpress.org/latest.tar.gz | sudo -u www-data tar zx -C /srv/www

    update wp-config.php with the new password

    sudo chown -R www-data:www-data /var/www/data
    find /var/www/data -type d -exec chmod 755 {} \;
    find /var/www/data -type f -exec chmod 644 {} \;

    Server setup

    Reverse nginx proxy settings

    HTTP Apache2 settings

  • Ubuntu basic commands

    Update packages/apps

    sudo apt update && sudo apt upgrade

    Compress/Decompress a file

    tar -zcvf compressedFileName.tar.gz directoryName

    tar -zxvf compressedFileName.tar.gz

    WSL use ssh key on remote server:
    eval ‘ssh-agent -s‘
    ssh-add -t 60m ~/.ssh/id_rsa
    ssh -A user:address

    MySQL

    sudo apt install mysql-server
sudo mysql_secure_installation
sudo /etc/init.d/mysql start
sudo service mysql restart

    For WSL automatic start after restart: 

    sudo update-rc.d mysql defaults

    SSH

    adduser roman
usermod -aG sudo roman
su - hideo
mkdir ~/.ssh && chmod 700 ~/.ssh
touch ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys
nano ~/.ssh/authorized_keys
sudo ufw app list
sudo ufw allow OpenSSH
sudo ufw enable
sudo ufw status
sudo apt update
sudo apt upgrade -u
sudo reboot

    nginX

    sudo nano /etc/nginx/nginx.conf
uncomment server_tokens off;

  • Win10 tools and settings for work with Ubuntu

    I spent a last few years working in Ubuntu and Linux Mint. Now I have a new computer and will do the setup with Win10 and WSL2.

    WSL settings and setup

    Turn Windows features on or off:

    • Telnet client
    • Virtual Machine Platform
    • Windows subsystem for linux

    Powershell: 

    wsl --set-default-version 2

    Tools and apps

    • Ubuntu – install from windows appstore
    • Windows terminal – install from windows appstore
    • Docker desktop – needed for running docker in linux subsystem
    • Dbeaver.io – for accessing databases
    • Visual Studio Code

    Ubuntu settings

    ssh key

    windows terminal -> settings -> starting directory /mnt/d/

    Trouble with accessing D: drive as a regular user
    create or edit file /etc/wsl.conf and change default username to yours:

    [user]
    default=roman

  • Git: Basic commands and scenarios

    If you use a repository on one of the version control platforms like github, gitlab or bitbucket then you need some basic commands.

    When I was first introduced to git many years ago, I had to start with git pull, add, commit and push. I use this bulk of commands till nowdays. These commands basically update your local version Additionally you will need git status to see where you are and what you want to add to the repository.

    Commands

    git pull – gets you the new code from the repository
    git add yourFile.php – add data for next commit
    git commit -m “message about your changes” – you can put custom message to your commit changes
    git push – upload commits to the repository

    Scenarios

    Join the existing project

    When you are invited to the existing project/repository you need to get the code to your local machine. You can find the the URL in the web gui of the repository and then you command git clone git@github.com:bcvsolutions/CzechIdMng.git <optional my folder name>

    Creating your own project

    Start with creating Git repository online first and use git clone to get it locally. Otherwise you have to make adjustments to connect it later. You will have to deal with existing files or master branch.

    Easy workaround

    If you already have the code and want to push it to a new repository. Then in my opinion the easiest way to make it happen is through new folder and copy the files and folders there. Rename your current folder to _old if needed, then do a git clone. Copy the files and folders from your _old folder to the repository and proceed with add, commit and push. In this way you don’t have to deal with main branch or other commands.

    Git init way

    If you have a newer version of git 2.28+ then you can use. New default branch name is “main” on github.

    git config --global init.defaultBranch main

    git init
    git add .
    git commit -m “message”
    git remote add origin <URL of your git repository>
    git push -u origin main – here could be master instead of main, if you happen to have it as a new default branch.

    Create a develop branch

    As soon as you start testing your code locally or on the server. Try to create a new branch called develop or staging as it will help you in the future to divide production and test environment.

    git checkout main – go to the branch main
    git pull
    git checkout -b develop – create and switch to branch develop from current branch(main) – this is is a shortcut instead of using git branch

    When you have properly tested your code and you ready to push the changes to a production environment. Merge develop to main branch. That way you will have a working code in the production and space for developing new features.

    git checkout main
    git pull
    git merge develop

    Remove changes from a file

    This restores completely the file — you will loose changes in staged area
    git checkout HEAD — myfile.java

    Forgot to commit a change, which will cause a new commit with minor detail

    git reset –soft HEAD~1
    git commit -m “changes together”