With By colour, the tool takes the most common colour along the image border as the background colour (the "key"), on the assumption that the border is background. Press Pick and click the image to choose a different spot; a 3 by 3 pixel area is averaged so JPEG noise does not decide the colour.
Each pixel is compared with the key using a colour distance weighted for how the eye sees red, green and blue, on a scale where black to white is 100. Pixels within the Tolerance (default 12) become fully transparent. Pixels slightly further away, within the Edge softness band (default 8), become partly transparent, which keeps edges smooth instead of jagged. Pixels right at the edge of the subject that blend its colour with the background are also made partly transparent, in proportion to how much background they contain, so anti-aliased outlines do not leave a halo. Setting Edge softness to 0 turns both off and gives hard edges, which suits pixel art.
By default only background that touches the image edge is removed: the tool starts at the border and spreads inwards through matching pixels, so the same colour inside the subject is kept. Tick Also remove enclosed areas to remove every matching pixel, for example the inside of a letter "O" in a logo. Remove the background colour from soft edges subtracts the old background from partly transparent edge pixels, so a cut-out does not keep a white or green outline.
The preview updates as you move the sliders, so you can see straight away what is removed. Zoom in on thin parts of the subject before downloading.