Although there are a number of unique stories from Maj. Ilfrey's service during the war, one of the most colorful took place on November 15, 1942. At the time, Ilfrey was part of the 1st Fighter Group, flying P-38's, stationed at Chivenor, in the Lands End region of England. It was on this day that the group flew from Chivenor to North Africa, as part of "Operation Torch", the invasion of North Africa. Not long after take-off, one of Ilfrey's long-range drop tanks departed, so he no longer had enough fuel to complete the flight as planned. He thought early on that he would still have enough fuel to land in Gibraltar, but that didn't prove so, and he was forced to land in Portugal. Being a neutral country, he was told immediately after he landed that he would be interned. He was asked, however, if one of the Portuguese pilots could come up on the plane to have a look in the cockpit, and Ilfrey agreed. As the pilot was sitting on the wing, Ilfrey started the engines up as a demonstration. As he was doing so, another P-38 was making an emergency landing, and Ilfrey took the opportunity in hand, advancing the throttles and blowing the Portuguese pilot off the wing. Ilfrey took off and finally made it to Gibraltar. Because of the international incident that had just occurred, the U.S. State Department in Washington was actually demanding for Ilfrey's return to Portugal (something that Ilfrey didn't know at the time). However, the Commander of American Opeations at Gibraltar wired Washington telling them that Ilfrey had already left for North Africa before the order had arrived from the State Department, and nothing more ever came of it.
Later on, with the 20th FG, in the summer of '44, Ilfrey was shot down over France, 200 miles behind enemy lines. Some French locals took him in and disguised him as a French farmer, Jacques Robert, and helped him to escape. Within just six days, Ilfrey was back in England once more. Although all pilots who evaded capture were supposed to be sent back to the U.S., Ilfrey was able to get around the rules and got right back into combat flying.