A recently introduced feature added to the popular communications app WhatsApp promised users they would be able to delete sent messages before they are seen may not actually delete the messages at all.

A number of Android users, including the Android Jefe blog, discovered WhatsApp doesn’t truly remove sent messages when the sender deletes them and recipients can easily track down the messages with just a little bit of tweaking.

Here’s how WhatsApp’s “unsend” feature, launched October 31, is supposed to work: When a user sends a message to a contact that they wanted to take back before it was read, they can tap and hold on the message. Doing so will produce a menu with “Delete” options. The user selects “Delete for everyone” and the message disappears from the conversation and can no longer be read by the intended recipient.

The feature did have a few limitations; the unsend feature would have be used within seven minutes of the message being sent, and the recipient would have to be running the latest version of WhatsApp that supports the message recall feature. But in theory, the message should be gone for good.

Unfortunately, that appears not to be the case. As with most things on the internet, it’s hard to truly remove something for good and WhatsApp messages do not appear to be the exception—at least on Android devices.

On Android, WhatsApp messages are stored in the device’s notifications list regardless if it was deleted in the app or not. Because of this feature, Android users can easily see at least part of a deleted message—notifications typically display about 100 or so characters.

That character limit does at least place a certain limitation on what a person can recover from a deleted message, and it is worth noting that it only preserves text and not multimedia messages. Users also won’t receive a notification for a message if they are interacting with someone else on WhatsApp when the deleted message was initially received.

While the ability to see at least part of a deleted message may cause problems for WhatsApp users, there is also a way to extend the clock and delete messages long after the seven minute window for deleting messages dictated by the app has expired.

WhatsApp bases the seven minute countdown on the system clock of the device that sent the message, so users can adjust the clock on their device to revert back to that seven minute time frame and remove a message after more than seven minutes have passed. The ability to roll back the clock works for at least a week after the message was originally sent. Just remember: those messages may not truly be deleted in full.

Despite the limitations on the ability to read deleted messages, the very fact that some part of a message that a person intended to delete can be recovered is problematic for users. It effectively means the only sure-fire way to know that a person will not see any part of a message is to just not send it.