BGSiteHolder

Failing HealthChecks on Regional Google Cloud TCP Load Balancer

Often Google are right, but sometimes not.

Recently I was looking at why GCP TCP Load Balancer healthchecks were failing, despite I have configured strictly my local OS firewall to allow the documented health check IP ranges :

(https://cloud.google.com/compute/docs/load-balancing/network/ – Section Firewall rules and Network load balancing  – 209.85.152.0/22209.85.204.0/22, and 35.191.0.0/16 at the time of writing).

Network Services–>Load Balancing–>Load Balancers tab was stating my instances were unhealthy:

Instance <instance name> is unhealthy for <LBIP>

It turned out Google is sending healthcheck from their metadata IP address instead  – metadata.google.internal (169.254.169.254)

Allowing this in the OS firewall made the HCs succeed!

Xpenology on VMware – Disk Extend

Recently I run out of free space on one of my XPenology DSM 6 drives and had to run an extend.

Since it’s running on ESXi, the VMware disk extend was easy. However surprisingly or not the Synology part wasn’t. The GUI does not allow you to increase the size of the RAID Group (in my case Basic without protection – single disk RAID1, Storage Manager–>RAID Group–>Manage the “Expand” option is grayed out)

So it was obvious manual resize through SSH is necessary.

After some time spend on googling good articles for doing this, the best one i found was http://blog.azib.net/2013/12/how-i-upgraded-my-synology-nas-to.html. It explains quite well the overall process of manually extend the volume (extend the vmware disk partition, the MD device, the physical, virtual and logical LVM volumes) , however I found Step 1 from the guide as dangerous (the partition recreation with a larger one) so I decided to try the parted partition resize option instead.

Below are the commands used, but first PLEASE BACKUP YOUR DATA without attempting anything:

  1. Extend the Disk in the VMware VM and connect to Synology using SSH (sudo su as needed).
  2. parted – launch the parted partition editor
  3. select /dev/sdX – chose which drive to work with (in my case /dev/sdc)
  4. unit s – work with sectors instead of GBs (otherwise you might get a message saying you get outside of disk boundaries)
  5. print free – two reasons for running this – 1. make sure you work with the correct drive; and 2. that you see the free space provisioned from VMwareModel: VMware Virtual disk (scsi)
    Disk /dev/sdc: 6012954214s
    Sector size (logical/physical): 512B/512B
    Partition Table: gpt
    Disk Flags:

    Number Start End Size File system Name Flags
    34s 2047s 2014s Free Space
    1 2048s 4982527s 4980480s ext4 raid
    2 4982528s 9176831s 4194304s linux-swap(v1) raid
    9176832s 9437183s 260352s Free Space
    3 9437184s 5153755935s 5144318752s raid
    5153755936s 6012954180s 859198245s Free Space      <–the free space is after the last (3rd) partition in the disk

  6. resizepart – run the command to resize the partition
    1. Partition number? 3 <– enter the partition number to edit (replace with 3 with your partition number)
    2. End?  [5153755935s]? 6012954180s <–Enter the new last partition sector as seen in the print free command
  7. quit the parted editor
  8.  Now it’s time to extend the RAID MD device. To do it first you have to stop any activity on it
    1. lsof | grep volume 3 (if lsof is missing – install it from here http://ipkg.nslu2-linux.org/feeds/optware/syno-i686/cross/stable/) <– check for open files on the volume
    2. In my case only samba was using the drive and it was stopped by /usr/syno/etc.defaults/rc.sysv/S80samba.sh stop .Follow Anton’s guide since it’s more detailed on how to find & stop the service using the drives, especially if you work with the volume where Synology packages are installed – usually volume1.
    3. lsof | grep volume3 <– run it again to ensure nothing remains using the volume.
    4. umount /volume3 <– unmount the volume (the GUI might start reporting it as crashed – ignore this for the moment)
    5. vgchange -a n vg3 <– deactivate the LVM Volume Group. Replace vg3 with your VolumeGroup Name (use vgdisplay -v or vgs commands to identify it)
    6. Stop and Re-assemble the RAID array
      mdadm -S /dev/md5 <– replace /dev/md5 with your MD device
      mdadm -A /dev/md5 -U devicesize /dev/sdc3 <–replace /dev/md5 with your MD device and /dev/sdc3 with your disk partition
    7. mdadm –grow /dev/md5 -z max <–extend the size of the MD array
    8. pvresize /dev/md5 <–extend the LVM Physical Volume and you’ll see the free space in the Physical [email protected]:/usr/syno/etc# vgs
      VG #PV #LV #SN Attr VSize VFree
      vg1 1 2 0 wz–n- 895.40g 0
      vg2 1 2 0 wz–n- 415.40g 0
      vg3 1 2 0 wz–n- 2.80t 409.70g
    9. vgchange -a y vg3 <– Activate the LVM volume group
    10. lvextend -L +409GB /dev/vg3/volume_3 <–extend the VLM Logical Volume to the last full GB
    11. lvextend -L +712MB /dev/vg3/volume_3 <–extend it with the remaining MBs as well(use vgs to see how much exact MBs are outstanding)
    12. [email protected]:/usr/syno/etc# vgs <–run vgs to validate you’ve used all available free space
      VG #PV #LV #SN Attr VSize VFree
      vg1 1 2 0 wz–n- 895.40g 0
      vg2 1 2 0 wz–n- 415.40g 0
      vg3 1 2 0 wz–n- 2.80t 0
    13. mount /volume3 <–mount the volume
  9. At this stage I expected the Synology GUI to reflect the actual RAID group and volume size, but it wasn’t the case.  After a Reboot (which starts the stopped services as well) the RAID group started showing the correct size after the extension. The Volume was still with the old size, but the GUI was handy (Login in Synology–>Storage Manager –>Volume –>select volume 3 –>expand) and the volume was extended, which ended the procedure.

And REMEMBER”

ALWAYS BACKUP YOUR DATA FIRST!

Enjoy!

Citrix XenServer hangs on detecting hardware during deployment on Hyper-V

Recently I had to deploy XenServer 6.X on my Hyper-V LAB on Windows Server 2012 R2 to evaluate a monitoring solution with 3rd party MS SCOM management pack.
The deployment was expected to be straightforward based on experience I have from other Linux distributions and VMware ESX. However during boot time XenServer was always hanging on “Detecting Hardware.”.

XenServer hangs on Detecting Hardware

Most of the articles over Internet says to disable ACPI, despite in XenServer 6.X the native safe mode has already that option turned on as well as others compatibility settings like noirqbalance, noapic, etc.

Installing Magento CE 2.0 on Windows Server 2012 R2 IIS 8.5

Magento (https://magento.com/) is one of the top 3 Open Source ecommerce platforms. Designed to run on Linux & Apache it could be a challenge to get it running on Windows IIS right from the first time since officially Magento is not supported on IIS.

Where to start from:
1. Deploy IIS 8.5 using Server Manager and add the SMTP server feature(or use an external SMTP server). It will be required later to send order emails from Magento
Install URL-rewrite IIS add on from http://www.iis.net/downloads/microsoft/url-rewrite
2. Install a MySQL server (or have a remote one)
3. Install php (5.5.x, 5.6.x or 7.0.2) http://devdocs.magento.com/guides/v2.0/install-gde/system-requirements.html and enable the required PHP modules as per the official magento documentation.
4. Download Magento https://www.magentocommerce.com/download , extract it in C:\inetpub\wwwroot (or a custom one ex. C:\Websites\Magento – something I recommend in general )
5. Configure php settings required by magento (in this example php 5.6.x was used)
The magento requirement says you have to configure these as a minimum http://devdocs.magento.com/guides/v2.0/install-gde/prereq/php-ubuntu.html#instgde-prereq-timezone
They are safe to be configured in you global php.ini file”

memory_limit = 768M
always_populate_raw_post_data = -1
asp_tags = Off
date.timezone = Europe/Sofia http://php.net/manual/en/timezones.php

Magento however customize some php settings using the .htaccess file. Since we’ll be using IIS these have to be converted in IIS aware setup (web.config). A good article how to configure php settings per IIS website can be found here http://www.iis.net/learn/application-frameworks/install-and-configure-php-on-iis/enable-per-site-php-configuration-on-iis-60-and-iis-7-and-above
Then you can find the cutomized PHP settings in the .htaccess file in the root magento directory
Edit your global php.ini file and add
user_ini.filename = “.user.ini”

Then create an .user.ini file in the root magento directory and copy the .htaccess php settings content for your PHP version. In our case the content for php 5.6 is:

memory_limit = 768M
max_execution_time = 18000
session.auto_start = Off
zlib.output_compression = On
suhosin.session.cryptua = Off

Create another .user.ini file inside pub directory and take the settings from pub\.htaccess:

memory_limit = 256M
max_execution_time = 18000
session.auto_start = Off
zlib.output_compression = Off
suhosin.session.cryptua = Off

6.Converting Magento Apache based .htaccess into IIS web.config
6.1 URL Rewrite Rules
You can use the URL-rewrite IIS settings to import .htaccess URL rewrite rules (URL Rewrite–>Import Rules–>Browse the file and select Import–>remove the non-convertable settings and click apply.
Alternatively place this in web.config into the magento root directory

<?xml version=”1.0″ encoding=”UTF-8″?>
<configuration>
<system.webServer>
<rewrite>
<rules>
<rule name=”Imported Rule 1″ stopProcessing=”true”>
<match url=”.*” ignoreCase=”false” />
<conditions logicalGrouping=”MatchAll”>
<add input=”{REQUEST_METHOD}” pattern=”^TRAC[EK]” ignoreCase=”false” />
</conditions>
<action type=”Redirect” url=”{R:0}” redirectType=”Temporary” />
</rule>
<rule name=”Imported Rule 2″ stopProcessing=”true”>
<match url=”^(.*)$” ignoreCase=”false” />
<conditions logicalGrouping=”MatchAll”>
<add input=”{URL}” pattern=”^/mobiledirectoryhere/.*$” ignoreCase=”false” negate=”true” />
<add input=”{HTTP_USER_AGENT}” pattern=”&quot;android|blackberry|ipad|iphone|ipod|iemobile|opera” ignoreCase=”false” />
</conditions>
<action type=”Redirect” url=”/mobiledirectoryhere/” redirectType=”Found” />
</rule>
<rule name=”Imported Rule 3″ stopProcessing=”true”>
<match url=”.*” ignoreCase=”false” />
<conditions logicalGrouping=”MatchAll”>
<add input=”{REQUEST_FILENAME}” matchType=”IsFile” ignoreCase=”false” negate=”true” />
<add input=”{REQUEST_FILENAME}” matchType=”IsDirectory” ignoreCase=”false” negate=”true” />
</conditions>
<action type=”Rewrite” url=”index.php” />
</rule>
</rules>
</rewrite>
</system.webServer>
</configuration>

Place this inside pub\media\web.config

<?xml version=”1.0″ encoding=”UTF-8″?>
<configuration>
<system.webServer>
<rewrite>
<rules>
<rule name=”Imported Rule 4″ stopProcessing=”true”>
<match url=”.*” ignoreCase=”false” />
<conditions logicalGrouping=”MatchAll”>
<add input=”{REQUEST_FILENAME}” matchType=”IsFile” ignoreCase=”false” negate=”true” />
</conditions>
<action type=”Rewrite” url=”../get.php” />
</rule>
</rules>
</rewrite>
</system.webServer>
</configuration>

Place this inside pub\static\web.config

<?xml version=”1.0″ encoding=”UTF-8″?>
<configuration>
<system.webServer>
<rewrite>
<rules>
<rule name=”Imported Rule 5″ stopProcessing=”true”>
<match url=”^version.+?/(.+)$” ignoreCase=”false” />
<action type=”Rewrite” url=”{R:1}” />
</rule>
<rule name=”Imported Rule 6″ stopProcessing=”true”>
<match url=”.*” ignoreCase=”false” />
<conditions logicalGrouping=”MatchAll”>
<add input=”{REQUEST_FILENAME}” matchType=”IsFile” ignoreCase=”false” negate=”true” />
</conditions>
<action type=”Rewrite” url=”../static.php?resource={R:0}” appendQueryString=”false” />
</rule>
</rules>
</rewrite>
</system.webServer>
</configuration>

Place this inside pub\web.config

<?xml version=”1.0″ encoding=”UTF-8″?>
<configuration>
<system.webServer>
<rewrite>
<rules>
<rule name=”Imported Rule 7″ stopProcessing=”true”>
<match url=”.*” ignoreCase=”false” />
<conditions logicalGrouping=”MatchAll”>
<add input=”{REQUEST_METHOD}” pattern=”^TRAC[EK]” ignoreCase=”false” />
</conditions>
<action type=”Redirect” url=”{R:0}” redirectType=”Temporary” />
</rule>
<rule name=”Imported Rule 8″ stopProcessing=”true”>
<match url=”.*” ignoreCase=”false” />
<conditions logicalGrouping=”MatchAll”>
<add input=”{REQUEST_FILENAME}” matchType=”IsFile” ignoreCase=”false” negate=”true” />
<add input=”{REQUEST_FILENAME}” matchType=”IsDirectory” ignoreCase=”false” negate=”true” />
</conditions>
<action type=”Rewrite” url=”index.php” />
</rule>
</rules>
</rewrite>
</system.webServer>
</configuration>

Make sure the rule name are unique in the deployment ! Otherwise you might get HTTP Error 500.52 – URL Rewrite Module Error


6.2 Convert the Mime Types

[Update Needed]

6.3 Convert all Remaining Settings

[Update Needed]

7. NTFS permissions
IIS APPPOOL\ApplicationPoolIdentity” (ex. “IIS APPPOOL\MagentoLive”) need to have :
READ&EXECUTE access on the magento directory (C:\Websites\Magento)
MODIFY access on magento \app\etc directory (for security purposes this should be removed after the installation is complete
MODIFY access on magento \var directory
MODIFY access on magento \pub\media directory
MODIFY access on magento \pub\static directory
MODIFY access on the PHP Temp directory  (otherwise it throws an errors like “We’re sorry, an error has occurred while generating this email” on the Storefront or “Could not determine temp directory, please specify a cache_dir manually” in the Admin panel
In case PHP Temp directory is not specified in php.ini then PHP takes the folder in the TEMP Environment variable (C:\Windows\Temp by default).

The Default ACL on C:\Windows\Temp on Windows 2012 R2 is (icacls output)
C:\Windows\Temp CREATOR OWNER:(OI)(CI)(IO)(F)
NT AUTHORITY\SYSTEM:(OI)(CI)(F)
BUILTIN\Administrators:(OI)(CI)(F)
BUILTIN\Users:(CI)(S,WD,AD,X)
BUILTIN\IIS_IUSRS:(OI)(CI)(S,RD)
In case magento is the single website on your IIS server you can directly grant the IIS_IUSERS MODIFY permissions
Important Security Notice: for all magento directories where you grant write access make sure you remove scripts execution in IIS (go to each folder–>Handled-Mappings–>Edit Feature Permissions–> uncheck Script)
8. Setup PHP OpCache (don’t miss this step! – it’s a great performance boost)

Edit you global php.ini and insert:

inside [php] section
zend_extension=php_opcache.dll (the DLL file must be present in your php extensions directory)
inside [opcache] section
opcache.enable=1
opcache.enable_cli=1
opcache.memory_consumption=128
opcache.interned_strings_buffer=8
opcache.max_accelerated_files=4000
opcache.use_cwd=1
opcache.revalidate_freq=60
opcache.fast_shutdown=1

9. Setup SSL on you IIS website – a lot of content exist on Internet how to do this. It doesn’t matter if you do this before or after the Magento Installation. It’s important to avoid using self-signed certificate.

10. Magento Installation
There are two ways to do this:
A) using the http://yoursite.com/setup web based wizard

or

B) using command line

cd magentorootdirectory 

php magento setup:install –admin-firstname John –admin-lastname Doe –admin-email [email protected] –admin-user admin –admin-password yourpassword –base-url http://yoursite.com –db-host=localhost –db-name DB –db-user DBuser –db-password yourDBpassword –use-rewrites 1 –use-secure-admin 1 –session-save db

List of support command line parameters can be found here http://devdocs.magento.com/guides/v2.0/install-gde/install/cli/install-cli-install.html

When you install from the website the Magento installation may hang on (freeze on) Installing schema.. or Upgrading schema.. usually around 4% or 5%.
This happens due to IIS FastCGI “Activity timeout” settings (by default 30 seconds)

Adjust the settings to a higher enough value to allow the setup to finish. ex. 3600 (1h)

Then retry the installation by :
deleting magentorootdirectory\app\etc\config.php and magentorootdirectory\app\etc\env.php
dropping the MySQL DB tables
re-running the setup from http://yoursite.com/setup
Until you retry the installation Magento website will return HTTP error 500 Internal Server Error

The installation might fail to deploy the static content making Storefront and the Admin page looking badly formatted(missing pictures, CSS styles, js, etc). To resolve this run:

“php bin\magento setup:static-content:deploy en_US”

http://devdocs.magento.com/guides/v2.0/install-gde/trouble/tshoot_no-styles.html
https://github.com/magento/magento2/issues/1312
When using non en-US Locale (Admin–>Stores–>Configuration–>General–>Locale Options) even after deploying static content, the StoreFront will still be badly formatted (Admin will be now OK). Two ways to solve this:
A) Set en-US locale in the Admin panel
B) Deploy static content for your locale

“php bin\magento setup:static-content:deploy bg_BG”

http://devdocs.magento.com/guides/v2.0/config-guide/cli/config-cli-subcommands-static-view.html

11. Magento Store Configuration

[More updates will follow]

Auto DoS feature on HP V1810 switches blocks legitimate network traffic

Luckily the default state of the Auto DoS feature on the HP V1810 switches is disabled. And you should always leave it like this.

It’s one of the worse implemented features ever seen on a managed switch. The idea for additional L2 security layer is good, however the realization is definitely poor. With the basic enabled/disabled states and zero customization available, the features set goes into the recycle bin. In particular it blocks all NetBIOS traffic over the LAN due to the “nice” feature called “Prevent UDP Blat Attack”.
It seems HP engineers are not aware NetBIOS use the same UDP Port 138 or maybe they believe it’s rarely used protocol nowadays.
However what happens when you replace some legacy switches on your network with a new HP V1810 series 48 port 1Gbps switches? What I’ve seen in particular is inability to join PCs in an Active Directory domain using the domain NetBIOS names with the following symptoms:
– Error 0x54b(1355 in decimal) present in %windir%\debug\NetSetup.log during PC join attempts.
– nltest /dsgetdc returns ERROR_NO_SUCH_DOMAIN (the same 1355/0x54b)
– no NetBIOS broadcast resolution
– no NetBIOS traffic reaching destination server with local LMHosts file entries enabled
– and much more.

Clearly reading the full vendor documentation before deploying any new equipment is a must. Otherwise you can spend hours in troubleshooting to understand how a single check box affects your Active Directory environment.

Reference: Auto Dos features description.
Auto DoS

Enable – Select to prevent receiving packets from the all attacks mentioned below (Default: Disabled).

Prevent Land Attack – Prevents receiving packets with matching Source and Destination IP addresses.
Prevent TCP Blat Attack – TCP Source and Destination Port match
Prevent UDP Blat Attack – UDP Source and Destination Port match
Prevent Ping Of Death Attack – Prevents receiving ping packets with a size larger than 512 bytes through the use of fragments, which can target vulnerable systems.
Prevent Invalid TCP Flags Attack – Prevents receiving packets with invalid TCP flags. TCP Flag SYN set and Source Port less than 1024 or TCP Control Flags = 0 and TCP Sequence Number = 0 or TCP Flags FIN, URG, and PSH set and TCP Sequence Number = 0 or TCP Flags SYN and FIN set.
Prevent TCP Fragment Attack – Drop IP Packets that have a TCP header less than 20 bytes.
Check First Fragment Only – Enable checking DOS attacks on IP first fragments
Prevent Smurf Attack – ICMP Echo packets (ping) to a broadcast IP address are dropped.
Prevent Ping Flood Attack – Prevents Ping Flood by limiting the number of ICMP Ping packets. The rate is 1000 ICMP packets per second.
Prevent Syn Flood Attack – A SYN flood attack sends TCP connections requests faster than a machine can process them. Setting this filter limits the rate of TCP connection requests.

Sluggish Hyper-V performance and Intel C-State Tech

A few weeks ago after an in-place upgrade from Windows 2012 to 2012 R2 Hyper-V started behaving poorly, especially during the VMs power on. All of them were spending abnormal duration on “Starting (10%)” phase – at least several seconds. Quite annoying problem which definitely required resolution as on the same box with win2k8 R2 Hyper-V worked like a charm – a VM used to power on in less than a second.

After troubleshooting through drivers compatibility and processor performance counters the root cause was found in the enabled Intel C-State Tech feature in BIOS. Disabling this power saving feature returned back the expected performance of Hyper-V.