Cryptography is the science of protecting secrets. As a result, it is designed to prevent an unauthorized party (such as a hacker) from accessing protected data. While early encryption algorithms had significant flaws and were easy to crack, the state of encryption has become much better.
That being said, cryptography can be broken under certain circumstances.
The first step to breaking cryptography is to determine the cryptographic algorithm being used. This can be done in a number of ways, including:
Manual reverse engineering
Keyword searches for algorithm names
Dependency analysis of the executable
Ciphertext analysis (such as observing the block length)
With this information in hand, it is time to start looking for a vulnerability that can be exploited.
Exploiting Cryptographic Vulnerabilities
Modern cryptography is secure by design. However, secure by design does not necessarily mean perfect security. If cryptography is poorly implemented or used incorrectly, vulnerabilities can be created that a hacker can exploit. There are many different ways to exploit modern cryptosystems.
Key Management
Encryption algorithms are designed to ensure that only the sender and receiver of a message can access it. This access is protected by secret keys that are known only to two authorized parties.
However, key management is one of the most common ways in which cryptography fails. Key management systems face the classic problem of balancing usability against security, and usability often wins. Encryption keys are often password-based or password-protected. Since many users commonly use weak or reused passwords, stealing an encryption key is often an effective way to bypass cryptographic protection.
Stream Cipher Vulnerabilities
Encryption algorithms come in two main varieties: block ciphers and stream ciphers. In general, block ciphers are more intuitive and easier to use, but less efficient (since you have to work with set block sizes). As a result, developers sometimes use stream ciphers.
While stream ciphers can certainly be secure, they also have their drawbacks. One potential problem is that stream ciphers perform encryption at the bit level. If the developer does not perform integrity checks, a hacker will not be able to read the protected data, but they can change it without detection.
Weak / Broken Encryption Algorithms
Another serious problem that exists with stream ciphers (as well as block ciphers) is the use of insecure encryption algorithms. RC4 is probably the stream cipher with the highest name recognition, and this name recognition means that it is used.
However, the developers who use it do not always know how to use it correctly. RC4 has a serious cryptographic weakness where the first bits of its output bitstream are highly correlated with the secret key. If the developer fails to exploit RC4 (i.e., by discarding these bits) and an attacker can guess the output bitstream (i.e., if they can choose the input to the cipher), they can obtain the encryption key, or at least enough bits, to make it vulnerable to a brute-force attack.
Despite its weaknesses, RC4 is still considered a usable cipher if used correctly. Other encryption algorithms (e.g., DES) are known to be vulnerable. Researching the vulnerabilities of encryption algorithms can allow a hacker to break the cryptography.
Custom Algorithms/Protocols
"Never start your own cryptocurrency" is essentially a law of cryptography. Expert cryptanalysts can spend years designing a cipher, and it can quickly be broken through a vulnerability they did not consider.
Despite this, people still believe that they can create their own algorithms that are better than the standard ones. If you can get a copy of a piece of software and it has a function with an excessive number of XOR operations and bit shifts, it is probably a custom algorithm. It is definitely worth the effort of reverse engineering, as it is almost certainly insecure.
Misuse of an Algorithm
Another common mistake developers make is misuse of algorithms. For example, asymmetric cryptography should always be used to encrypt traffic between a server and a client. Otherwise, the encryption key is embedded somewhere in the client code. If you believe that the code in question uses symmetric encryption to communicate, it is time to look for that key in the executable.
Misuse of an algorithm also includes many other errors. Reuse of a key
which can be a serious problem for certain algorithms, so look for a function that generates a shared key between the client and server. If it doesn't exist, the algorithm may be vulnerable to attack. If it does exist, a man-in-the-middle attack may be able to decrypt the traffic.
Algorithms can also be misused due to the incorrect use of certain values. If an encryption algorithm:
Hashes the password without a unique salt
Uses a non-random or non-unique initialization vector
Then it is likely vulnerable to attack. Finding the specific algorithm and how it affects its security may be the key to breaking it.
Side Channels
Side channel analysis is an attack on cryptography in which there are unintentional data leaks due to the way the system is implemented. For example, the power consumption of some cryptographic algorithms depends in some way on the secret key.
Trying to measure the power consumption of an algorithm may not be possible, but execution time is another common side channel. The execution time of some algorithms depends on their input.
A non-cryptographic example is an authentication system that checks a password character by character and returns immediately if the password is found to be incorrect. This type of system can be circumvented by guessing the password one character at a time and monitoring the execution time. The character that takes the longest to process is likely the correct one for that bit of the password. If an encryption algorithm has similar timing leaks, it may be vulnerable to attack.
When performing side-channel analysis, it is important to consider all possible causes of the leak. For example, the Heartbleed vulnerability is caused by memory caching, which creates a timing side channel.
Out of scope
Finally, encryption algorithms are only effective at protecting data while it is encrypted. Encryption cannot protect data before encryption is performed or after decryption. If important data is encrypted at rest or in transit, it is often better to try to steal the data while it is unencrypted by monitoring the memory space of the process performing the encryption and any calls to cryptographic libraries.
Conclusion: Breaking Cryptography
Ultimately, cryptography can only be broken if it is done incorrectly. Modern encryption algorithms are designed to make guessing the secret key the most effective attack vector and to make it computationally infeasible on modern hardware.
However, encryption algorithms are also extremely sensitive to errors in design or implementation. Even a tiny error can make an algorithm vulnerable to attack.
That being said, cryptography can be broken under certain circumstances.
The first step to breaking cryptography is to determine the cryptographic algorithm being used. This can be done in a number of ways, including:
Manual reverse engineering
Keyword searches for algorithm names
Dependency analysis of the executable
Ciphertext analysis (such as observing the block length)
With this information in hand, it is time to start looking for a vulnerability that can be exploited.
Exploiting Cryptographic Vulnerabilities
Modern cryptography is secure by design. However, secure by design does not necessarily mean perfect security. If cryptography is poorly implemented or used incorrectly, vulnerabilities can be created that a hacker can exploit. There are many different ways to exploit modern cryptosystems.
Key Management
Encryption algorithms are designed to ensure that only the sender and receiver of a message can access it. This access is protected by secret keys that are known only to two authorized parties.
However, key management is one of the most common ways in which cryptography fails. Key management systems face the classic problem of balancing usability against security, and usability often wins. Encryption keys are often password-based or password-protected. Since many users commonly use weak or reused passwords, stealing an encryption key is often an effective way to bypass cryptographic protection.
Stream Cipher Vulnerabilities
Encryption algorithms come in two main varieties: block ciphers and stream ciphers. In general, block ciphers are more intuitive and easier to use, but less efficient (since you have to work with set block sizes). As a result, developers sometimes use stream ciphers.
While stream ciphers can certainly be secure, they also have their drawbacks. One potential problem is that stream ciphers perform encryption at the bit level. If the developer does not perform integrity checks, a hacker will not be able to read the protected data, but they can change it without detection.
Weak / Broken Encryption Algorithms
Another serious problem that exists with stream ciphers (as well as block ciphers) is the use of insecure encryption algorithms. RC4 is probably the stream cipher with the highest name recognition, and this name recognition means that it is used.
However, the developers who use it do not always know how to use it correctly. RC4 has a serious cryptographic weakness where the first bits of its output bitstream are highly correlated with the secret key. If the developer fails to exploit RC4 (i.e., by discarding these bits) and an attacker can guess the output bitstream (i.e., if they can choose the input to the cipher), they can obtain the encryption key, or at least enough bits, to make it vulnerable to a brute-force attack.
Despite its weaknesses, RC4 is still considered a usable cipher if used correctly. Other encryption algorithms (e.g., DES) are known to be vulnerable. Researching the vulnerabilities of encryption algorithms can allow a hacker to break the cryptography.
Custom Algorithms/Protocols
"Never start your own cryptocurrency" is essentially a law of cryptography. Expert cryptanalysts can spend years designing a cipher, and it can quickly be broken through a vulnerability they did not consider.
Despite this, people still believe that they can create their own algorithms that are better than the standard ones. If you can get a copy of a piece of software and it has a function with an excessive number of XOR operations and bit shifts, it is probably a custom algorithm. It is definitely worth the effort of reverse engineering, as it is almost certainly insecure.
Misuse of an Algorithm
Another common mistake developers make is misuse of algorithms. For example, asymmetric cryptography should always be used to encrypt traffic between a server and a client. Otherwise, the encryption key is embedded somewhere in the client code. If you believe that the code in question uses symmetric encryption to communicate, it is time to look for that key in the executable.
Misuse of an algorithm also includes many other errors. Reuse of a key
which can be a serious problem for certain algorithms, so look for a function that generates a shared key between the client and server. If it doesn't exist, the algorithm may be vulnerable to attack. If it does exist, a man-in-the-middle attack may be able to decrypt the traffic.
Algorithms can also be misused due to the incorrect use of certain values. If an encryption algorithm:
Hashes the password without a unique salt
Uses a non-random or non-unique initialization vector
Then it is likely vulnerable to attack. Finding the specific algorithm and how it affects its security may be the key to breaking it.
Side Channels
Side channel analysis is an attack on cryptography in which there are unintentional data leaks due to the way the system is implemented. For example, the power consumption of some cryptographic algorithms depends in some way on the secret key.
Trying to measure the power consumption of an algorithm may not be possible, but execution time is another common side channel. The execution time of some algorithms depends on their input.
A non-cryptographic example is an authentication system that checks a password character by character and returns immediately if the password is found to be incorrect. This type of system can be circumvented by guessing the password one character at a time and monitoring the execution time. The character that takes the longest to process is likely the correct one for that bit of the password. If an encryption algorithm has similar timing leaks, it may be vulnerable to attack.
When performing side-channel analysis, it is important to consider all possible causes of the leak. For example, the Heartbleed vulnerability is caused by memory caching, which creates a timing side channel.
Out of scope
Finally, encryption algorithms are only effective at protecting data while it is encrypted. Encryption cannot protect data before encryption is performed or after decryption. If important data is encrypted at rest or in transit, it is often better to try to steal the data while it is unencrypted by monitoring the memory space of the process performing the encryption and any calls to cryptographic libraries.
Conclusion: Breaking Cryptography
Ultimately, cryptography can only be broken if it is done incorrectly. Modern encryption algorithms are designed to make guessing the secret key the most effective attack vector and to make it computationally infeasible on modern hardware.
However, encryption algorithms are also extremely sensitive to errors in design or implementation. Even a tiny error can make an algorithm vulnerable to attack.
