Fermat's factorization method

For Fermat's method on determining extreme values, see Interior extremum theorem.

Fermat's factorization method, named after Pierre de Fermat, is based on the representation of an odd integer as the difference of two squares:

${\displaystyle N=a^{2}-b^{2}.}$

That difference is algebraically factorable as ${\displaystyle (a+b)(a-b)}$; if neither factor equals one, it is a proper factorization of N.

Each odd number has such a representation. Indeed, if ${\displaystyle N=cd}$ is a factorization of N, then

${\displaystyle N=\left({\frac {c+d}{2}}\right)^{2}-\left({\frac {c-d}{2}}\right)^{2}.}$

Since N is odd, then c and d are also odd, so those halves are integers. (A multiple of four is also a difference of squares: let c and d be even.)

In its simplest form, Fermat's method might be even slower than trial division (worst case). Nonetheless, the combination of trial division and Fermat's is more effective than either by itself.

Basic method

One tries various values of a, hoping that ${\displaystyle a^{2}-N=b^{2}}$, a square.

FermatFactor(N): // N should be odd
    a ← ceiling(sqrt(N))
    b2 ← a*a - N
    repeat until b2 is a square:
        a ← a + 1
        b2 ← a*a - N 
     // equivalently: 
     // b2 ← b2 + 2*a + 1 
     // a ← a + 1
    return a - sqrt(b2) // or a + sqrt(b2)

For example, to factor ${\displaystyle N=5959}$, the first try for a is the square root of 5959 rounded up to the next integer, which is 78. Then ${\displaystyle b^{2}=78^{2}-5959=125}$. Since 125 is not a square, a second try is made by increasing the value of a by 1. The second attempt also fails, because 282 is again not a square.

Try:	1	2	3
a	78	79	80
b2	125	282	441
b	11.18	16.79	21

The third try produces the perfect square of 441. Thus, ${\displaystyle a=80}$, ${\displaystyle b=21}$, and the factors of 5959 are ${\displaystyle a-b=59}$ and ${\displaystyle a+b=101}$.

Suppose N has more than two prime factors. That procedure first finds the factorization with the least values of a and b. That is, ${\displaystyle a+b}$ is the smallest factor ≥ the square-root of N, and so ${\displaystyle a-b=N/(a+b)}$ is the largest factor ≤ root-N. If the procedure finds ${\displaystyle N=1\cdot N}$, that shows that N is prime.

For ${\displaystyle N=cd}$, let c be the largest subroot factor. ${\displaystyle a=(c+d)/2}$, so the number of steps is approximately ${\displaystyle (c+d)/2-{\sqrt {N}}=({\sqrt {d}}-{\sqrt {c}})^{2}/2=({\sqrt {N}}-c)^{2}/2c}$.

If N is prime (so that ${\displaystyle c=1}$), one needs ${\displaystyle O(N)}$ steps. This is a bad way to prove primality. But if N has a factor close to its square root, the method works quickly. More precisely, if c differs less than ${\displaystyle {\left(4N\right)}^{1/4}}$ from ${\displaystyle {\sqrt {N}}}$, the method requires only one step; this is independent of the size of N.^{[citation needed]}

Fermat's and trial division

Consider trying to factor the prime number N = 2,345,678,917, but also compute b and a − b throughout. Going up from ${\displaystyle {\sqrt {N}}}$ rounded up to the next integer, which is 48,433, we can tabulate:

Try	1st	2nd	3rd	4th
a	48,433	48,434	48,435	48,436
b2	76,572	173,439	270,308	367,179
b	276.7	416.5	519.9	605.9
a − b	48,156.3	48,017.5	47,915.1	47,830.1

In practice, one wouldn't bother with that last row until b is an integer. But observe that if N had a subroot factor above ${\displaystyle a-b=47830.1}$, Fermat's method would have found it already.

Trial division would normally try up to 48,432; but after only four Fermat steps, we need only divide up to 47830, to find a factor or prove primality.

This all suggests a combined factoring method. Choose some bound ${\displaystyle a_{\mathrm {max} }>{\sqrt {N}}}$; use Fermat's method for factors between ${\displaystyle {\sqrt {N}}}$ and ${\displaystyle a_{\mathrm {max} }}$. This gives a bound for trial division which is ${\displaystyle a_{\mathrm {max} }-{\sqrt {a_{\mathrm {max} }^{2}-N}}}$. In the above example, with ${\displaystyle a_{\mathrm {max} }=48436}$ the bound for trial division is 47830. A reasonable choice could be ${\displaystyle a_{\mathrm {max} }=55000}$ giving a bound of 28937.

In this regard, Fermat's method gives diminishing returns. One would surely stop before this point:

a	60,001	60,002
b2	1,254,441,084	1,254,561,087
b	35,418.1	35,419.8
a − b	24,582.9	24,582.2

Sieve improvement

When considering the table for ${\displaystyle N=2345678917}$, one can quickly tell that none of the values of ${\displaystyle b^{2}}$ are squares:

a	48,433	48,434	48,435	48,436
b2	76,572	173,439	270,308	367,179
b	276.7	416.5	519.9	605.9

It is not necessary to compute all the square-roots of ${\displaystyle a^{2}-N}$, nor even examine all the values for a. Squares are always congruent to 0, 1, 4, 5, 9, 16 modulo 20. The values repeat with each increase of a by 10. In this example, N is 17 mod 20, so subtracting 17 mod 20 (or adding 3), ${\displaystyle a^{2}-N}$ produces 3, 4, 7, 8, 12, and 19 modulo 20 for these values. It is apparent that only the 4 from this list can be a square. Thus, ${\displaystyle a^{2}}$ must be 1 mod 20, which means that a is 1, 9, 11 or 19 mod 20; it will produce a ${\displaystyle b^{2}}$ which ends in 4 mod 20 and, if square, b will end in 2 or 8 mod 10.

This can be performed with any modulus. Using the same ${\displaystyle N=2345678917}$,

modulo 16:	Squares are	0, 1, 4, or 9
	N mod 16 is	5
	so ${\displaystyle a^{2}}$ can only be	9
	and a must be	3 or 5 or 11 or 13 modulo 16
modulo 9:	Squares are	0, 1, 4, or 7
	N mod 9 is	7
	so ${\displaystyle a^{2}}$ can only be	7
	and a must be	4 or 5 modulo 9

One generally chooses a power of a different prime for each modulus.

Given a sequence of a-values (start, end, and step) and a modulus, one can proceed thus:

FermatSieve(N, astart, aend, astep, modulus)
    a ← astart
    do modulus times:
        b2 ← a*a - N
        if b2 is a square, modulo modulus:
            FermatSieve(N, a, aend, astep * modulus, NextModulus)
        endif
        a ← a + astep
    enddo

But the recursion is stopped when few a-values remain; that is, when (aend-astart)/astep is small. Also, because a's step-size is constant, one can compute successive b2's with additions.

Optimal a m a x {\displaystyle a_{\mathrm {max} }}

Premise

An optimal ${\displaystyle a_{\mathrm {max} }}$ can be computed using derivative methods.

The cost of executing Fermat’s method from ${\displaystyle {\sqrt {N}}}$ up to ${\displaystyle a_{\mathrm {max} }}$ is roughly proportional to a constant we will call ${\displaystyle d}$. Using sieving we can reduce it by some constant we call ${\displaystyle l}$. In the combined method the trial division bound becomes ${\displaystyle a_{\mathrm {max} }-{\sqrt {a_{\mathrm {max} }^{2}-N}}}$. Writing ${\displaystyle a_{\mathrm {max} }={\sqrt {N}}+d}$, one gets:

${\displaystyle a_{\mathrm {max} }^{2}-N=({\sqrt {N}}+d)^{2}-N={\sqrt {N}}^{2}+2{\sqrt {N}}d+d^{2}-N=2{\sqrt {N}}d+d^{2}}$

Substitute the new formula, we get

${\displaystyle a_{\mathrm {max} }-{\sqrt {a_{\mathrm {max} }^{2}-N}}\to a_{\mathrm {max} }-{\sqrt {2{\sqrt {N}}d+d^{2}}}}$

The goal is to choose a ${\displaystyle a_{\mathrm {max} }}$ such that ${\displaystyle C\left(d,N,l\right)={\frac {d}{l}}+\left(a_{\mathrm {max} }-{\sqrt {2{\sqrt {N}}d+d^{2}}}\right)\to d{\frac {1}{l}}+\left({\sqrt {N}}+d\right)-{\sqrt {2{\sqrt {N}}d+d^{2}}}={\sqrt {N}}+d\left(1+{\frac {1}{l}}\right)-{\sqrt {2{\sqrt {N}}d+d^{2}}}}$ is minimized.

Finding the Optimum

Differentiate ${\displaystyle C\left(d,N,l\right)}$ with respect to ${\displaystyle d}$. Due to the linearity of derivatives

${\displaystyle {\frac {d}{dd}}{\sqrt {N}}+d\left(1+{\frac {1}{l}}\right)-{\sqrt {2{\sqrt {N}}d+d^{2}}}={\frac {d}{dd}}{\sqrt {N}}+{\frac {d}{dd}}d\left(1+{\frac {1}{l}}\right)-{\frac {d}{dd}}{\sqrt {2{\sqrt {N}}d+d^{2}}}}$

Because ${\displaystyle {\sqrt {N}}}$ doesn't depend on ${\displaystyle d}$ we can drop that derivative term

for the ${\displaystyle d\left(1+{\frac {1}{l}}\right)}$ term, use the multiple rule ${\displaystyle \left(fg\right)'=f'g+g'f}$:

${\displaystyle \left(d\left(1+{\frac {1}{l}}\right)\right)'=d'\left(1+{\frac {1}{l}}\right)+\left(1+{\frac {1}{l}}\right)'d=\left(1+{\frac {1}{l}}\right)}$

For the last term, we use the chain rule ${\displaystyle \left(f\left(g\right)\right)'=f'\left(g\right)g'}$ and the power rule ${\displaystyle x^{n}=nx^{n-1}}$

${\displaystyle {\frac {d}{dd}}{\sqrt {2{\sqrt {N}}d+d^{2}}}={\frac {\left(2{\sqrt {N}}d+d^{2}\right)^{-{\frac {1}{2}}}}{2}}{\frac {d}{dd}}2{\sqrt {N}}d+d^{2}={\frac {\left(2{\sqrt {N}}d+d^{2}\right)^{-{\frac {1}{2}}}}{2}}\left(2{\sqrt {N}}+2d\right)=\left(2{\sqrt {N}}d+d^{2}\right)^{-{\frac {1}{2}}}\left({\sqrt {N}}+d\right)}$Substitute the known derivate formulas

${\displaystyle {\frac {d}{dd}}{\sqrt {N}}+{\frac {d}{dd}}d\left(1+{\frac {1}{l}}\right)-{\frac {d}{dd}}{\sqrt {2{\sqrt {N}}d+d^{2}}}=\left(1+{\frac {1}{l}}\right)-\left(2{\sqrt {N}}d+d^{2}\right)^{-{\frac {1}{2}}}\left({\sqrt {N}}+d\right)}$

To find the minimum, notice at the minimum the derivative vanishes, so st the derivative to 0

${\displaystyle \left(2{\sqrt {N}}d+d^{2}\right)^{-{\frac {1}{2}}}\left({\sqrt {N}}+d\right)=0\to {\frac {{\sqrt {N}}+d}{\sqrt {2{\sqrt {N}}d+d^{2}}}}=1+{\frac {1}{l}}}$

Square both sides to remove the root then cross multiply

${\displaystyle \left({\frac {{\sqrt {N}}+d}{\sqrt {2{\sqrt {N}}d+d^{2}}}}\right)^{2}=\left(1+{\frac {1}{l}}\right)^{2}\to {\frac {\left({\sqrt {N}}+d\right)^{2}}{2{\sqrt {N}}d+d^{2}}}=\left(1+{\frac {1}{l}}\right)^{2}\to \left({\sqrt {N}}+d\right)^{2}=\left(2{\sqrt {N}}d+d^{2}\right)\left(1+{\frac {1}{l}}\right)^{2}}$

Expand LHS

${\displaystyle \left({\sqrt {N}}+d\right)^{2}=\left(2{\sqrt {N}}d+d^{2}\right)\left(1+{\frac {1}{l}}\right)^{2}\to N+2{\sqrt {N}}d+d^{2}=\left(2{\sqrt {N}}d+d^{2}\right)\left(1+{\frac {1}{l}}\right)^{2}}$

Bring the right side to the left, Factor the common factor, Then, bring the second term to the right-hand side

${\displaystyle N+2{\sqrt {N}}d+d^{2}=\left(2{\sqrt {N}}d+d^{2}\right)\left(1+{\frac {1}{l}}\right)^{2}\to N=\left(2{\sqrt {N}}d+d^{2}\right)\left[\left(1+{\frac {1}{l}}\right)^{2}-1\right]}$

Simplify the bracket

${\displaystyle \left(1+{\frac {1}{l}}\right)^{2}-1=1^{2}+2\times 1\times {\frac {1}{l}}+\left({\frac {1}{l}}\right)^{2}-1={\frac {2}{l}}+{\frac {1}{l^{2}}}={\frac {2l+1}{l^{2}}}}$

So, the equation is now

${\displaystyle N=\left(2{\sqrt {N}}d+d^{2}\right){\frac {2l+1}{l^{2}}}\to Nl^{2}=\left(2{\sqrt {N}}d+d^{2}\right)\left(2l+1\right)}$

To apply the quadratic formula to solve ${\displaystyle d}$ we have to rewrite the equation to a quadratic. Write the right side as:

${\displaystyle \left(2{\sqrt {N}}d+d^{2}\right)\left(2l+1\right)\to \left(2l+1\right)2{\sqrt {N}}d+\left(2l+1\right)d^{2}}$

Since ${\displaystyle 2l+1\neq 0}$, you could divide through by it to get

${\displaystyle d^{2}+2{\sqrt {N}}d-{\frac {Nl^{2}}{2l+1}}=0}$

This is the quadratic equation we been looking for, we can now apply the quardratic formula:

${\displaystyle d={\frac {-2{\sqrt {N}}\pm {\sqrt {\left(2{\sqrt {N}}\right)^{2}-4\times 1\times \left(-{\frac {Nl^{2}}{2l+1}}\right)}}}{2}}\to -{\sqrt {N}}\pm {\sqrt {N}}{\frac {l+1}{\sqrt {2l+1}}}}$

Since ${\displaystyle d>0}$ we take the positive solution. Since ${\displaystyle a_{\mathrm {max} }={\sqrt {N}}+d}$ one gets:

${\displaystyle a_{\mathrm {max} }={\sqrt {N}}+d\to {\sqrt {N}}+-{\sqrt {N}}+{\sqrt {N}}{\frac {l+1}{\sqrt {2l+1}}}={\sqrt {N}}{\frac {l+1}{\sqrt {2l+1}}}}$

Cost

Substitute our optimal ${\displaystyle d}$ into ${\displaystyle C\left(d,N,l\right)}$

${\displaystyle {\sqrt {N}}+d\left(1+{\frac {1}{l}}\right)-{\sqrt {2{\sqrt {N}}d+d^{2}}}\to {\sqrt {N}}+\left(-{\sqrt {N}}+{\sqrt {N}}{\frac {l+1}{\sqrt {2l+1}}}\right)\left(1+{\frac {1}{l}}\right)-{\sqrt {2{\sqrt {N}}\left(-{\sqrt {N}}+{\sqrt {N}}{\frac {l+1}{\sqrt {2l+1}}}\right)+\left(-{\sqrt {N}}+{\sqrt {N}}{\frac {l+1}{\sqrt {2l+1}}}\right)^{2}}}}$

Simplifying the monster of an equation, we get ${\displaystyle {\frac {{\sqrt {N}}\left({\sqrt {2l+1}}-1\right)}{l}}}$.

Facts

• If ${\displaystyle l=1}$ that means no sieving, ${\displaystyle a_{\mathrm {max} }={\frac {2{\sqrt {N}}}{3}}}$ and the cost becomes ${\displaystyle {\sqrt {N}}\left({\sqrt {3}}-1\right)}$, which is still better than pure trial division or pure Fermat
• The ${\displaystyle a_{\mathrm {max} }-{\sqrt {a_{\mathrm {max} }^{2}-N}}}$ which is the trial division bound becomes ${\displaystyle {\frac {\sqrt {N}}{\sqrt {2l+1}}}}$ when subsututing the optimal.

Example

Using the same ${\displaystyle N=2345678917}$, if there's no sieving then you should choose ${\displaystyle a_{\mathrm {max} }}$ around 55924.69838392813, the reasonable choice ${\displaystyle a_{\mathrm {max} }=55000}$ is not that far off from the optimal with a bound of 28937, but the optimal choice gets a bound of 27962. If we are sieving modulo 20, then ${\displaystyle l={\frac {1}{\frac {4}{20}}}=5}$ and you should choose around ${\displaystyle {\sqrt {2345678917}}{\frac {6}{\sqrt {11}}}\approx 87617.1636423325}$ and this should intuitively make sense. If the Fermat part costs less, the spend more time in the Fermat part to lower ${\displaystyle a_{\mathrm {max} }-{\sqrt {a_{\mathrm {max} }^{2}-N}}}$

Multiplier improvement

Fermat's method works best when there is a factor near the square-root of N.

If the approximate ratio of two factors (${\displaystyle d/c}$) is known, then a rational number ${\displaystyle v/u}$ can be picked near that value. ${\displaystyle Nuv=cv\cdot du}$, and Fermat's method, applied to Nuv, will find the factors ${\displaystyle cv}$ and ${\displaystyle du}$ quickly. Then ${\displaystyle \gcd(N,cv)=c}$ and ${\displaystyle \gcd(N,du)=d}$. (Unless c divides u or d divides v.)

Generally, if the ratio is not known, various ${\displaystyle u/v}$ values can be tried, and try to factor each resulting Nuv. R. Lehman devised a systematic way to do this, so that Fermat's plus trial division can factor N in ${\displaystyle O(N^{1/3})}$ time.^[1]

Other improvements

The fundamental ideas of Fermat's factorization method are the basis of the quadratic sieve and general number field sieve, the best-known algorithms for factoring large semiprimes, which are the "worst-case". The primary improvement that quadratic sieve makes over Fermat's factorization method is that instead of simply finding a square in the sequence of ${\displaystyle a^{2}-n}$, it finds a subset of elements of this sequence whose product is a square, and it does this in a highly efficient manner. The end result is the same: a difference of squares mod n that, if nontrivial, can be used to factor n.

See also

• Completing the square
• Factorization of polynomials
• Factor theorem
• FOIL rule
• Monoid factorisation
• Pascal's triangle
• Prime factor
• Factorization
• Euler's factorization method
• Integer factorization
• Program synthesis
• Table of Gaussian integer factorizations
• Unique factorization