🔒 Locked

C++ virtual template function alternative

Started by popsoftheyear Feb 20, 2009 at 9:54 AM 16 replies 21.2k views

Original Post

Feb 20, 2009 at 9:54 AM

Unless I missed out on some sort of better design, I have a case (which I imagine is typical?) where there is a base abstract class IFile, and from it is derived classes such as FileIStream, FileMem, and FileDisk. With type safety in mind, I have this bright idea to implement a function like: // Wish this could be virtual and abstract :/ template void WriteArray(const std::vector &Buffer); Of course this does no good in the IFile interface, and although it is pretty useful in the concrete classes, when I actually need to depend on the interface alone, I'm stuck. What can I do so that I have the equivalent of a virtual template function? I read something on a curiously recurring template, but that would require that the base class interface knows about the derived... and if that were the case I wouldn't need to use the base class interface in the first place :/ The only thing I can think of is to implement the template function for the base class, and call a private function along the lines of: // Ewwww (but maybe good enough?) virtual void WriteArrayImpl(void *Mem, UInt64 Bytes) = 0; Any better ideas? Cheers -Scott

Rewdew

Feb 20, 2009 at 10:07 AM

The only thing you can think of is what I'd do (have the templated methods in IFile and have them call a protected/private pure virtual WriteArrayImpl)

kornman00

Feb 20, 2009 at 10:18 AM

Make an interface called "IStreamable" or what have you which your objects implement, then you can just iterate thru the vector and call the IStreamable's methods for reading\writing it's stream data to the IFile's implementation stream. It would be left up to the object to define the way the data is streamed.

I don't consider it typical for the Stream classes to provide that kind of detail, it should be up to the object's (proper OOP). Unless you're wanting to stream vector kind of stuff, then just have a template function (wouldn't be virtual) that iterates the vector and calls the Read\Write methods for that core type (ie, float). Then for vectors of IStreamable you can just iterate and once again, call the methods for reading\writing that they disclose.

Take a look at .NET's IO classes (yes I know this is C++). They may provide a better understanding of how to neatly go about doing this in your class interface design.

Bregma

Feb 20, 2009 at 10:31 AM

Quote:
Original post by popsoftheyear
Any better ideas?

Could you just do what the standard library does when it implements its streams? I mean, after all, they solved this problem 15 or 20 years ago and have had their solution reviewed by tens of thousands of peers over the years. Perhaps there is reason in their madness.

With a clean separation of interface and implementation, there is just no need for virtual abstract templated functions at all.

Antheus

Feb 20, 2009 at 10:33 AM

In general, templates and virtuals are same thing, but they can't be mixed. I'm not aware of any real need for virtuals that cannot be solved in a different way.

This type of problem is commonly solved with overloading:

template < class File, class T >void write(File & f, T & data);template < class File, class T, class A>void write(File & f, std::vector<T, A> & data);

and so on...

This allows you to specialize for specific files, as well as specific T:

template < class T >void write(NetworkFile & f, T & data) {  f.openConnection();  f.send(data);  f.close();}

Or, if you want to minimize the need for overloading, you can write a generic File something like this:

struct File {  template < class T >  void writeNumber(T & t);  template < class T, class A >  void writeVector(std::vector<T,A> & v);  template < class T >  void writeBytes(T * data, size_t n);};template < class File > void write(File & f, int i);template < class File > void write(File & f, unsigned int i);template < class File > void write(File & f, char i);template < class File > void write(File & f, char i);...// they all call writeNumber

This defines all valid types. Everything else is then expressed using those:

template < class File >void writeString(File & f, std::string & s) {  write(f, s.size());  write(f, s.c_str());}

And so on...

Once you get your head around how to specialize, and you define the fundamental types, as well as generalized write function for containers, the rest becomes completely trivial.

Note that you don't need for File classes to be related, or have any virtual methods.

If at some point you do need run-time polymorphism, then you still need to specialize for each File. This is often better solved with moving abstraction one level higher, so that instead you pass destination, and let implementation create writer/reader on its own.

rip-off

Feb 20, 2009 at 10:33 AM

I'm not even sure you need to design these classes. The standard C++ stream objects provide much of the functionality and are extensible if you need more.

Can you give an example of something you are trying to do with this that cannot be achieved using the standard stream objects?

popsoftheyear

Feb 20, 2009 at 10:45 AM

Ok. So bear with my while I try to get this. Say I have some class to load images. These images should be able to load, by calling the same interface, from an IStream, a disk file, or memory. Which one is something that isn't known until run-time. Is this something I can do with C++ streams? I was under the impression they would face the same exactly problems I'm facing, since they still end up being defined functions (the stream operators).

For instance, in Antheus's example, everything there assumes knowledge at compile time of the type of "File object". That's the whole point... I (essentially) *need* an abstract interface, but I *want* the template functions for writing arrays and specific types and what not. Those 2 don't seem to be mixing very well.

How is it that streams would fix this issue?

And thanks for all answers thus far :)

-Scott

EDIT:

Quote:
Antheus

If at some point you do need run-time polymorphism, then you still need to specialize for each File. This is often better solved with moving abstraction one level higher, so that instead you pass destination, and let implementation create writer/reader on its own.

How would it do that without the various File classes inheriting from a common base at the very least? Aside from some flag and a switch/case for every function...

I mean I've done this before but every class type inherited from a common interface so that I just had to dynamic cast a bunch of times until I found which it was, and used that instantiation from there on out.

SiCrane

Feb 20, 2009 at 11:03 AM

Just implement your image loader in terms of std::istream, and then use different streambufs for your different sources.

Antheus

Feb 20, 2009 at 11:04 AM

Quote:
Original post by popsoftheyear

For instance, in Antheus's example, everything there assumes knowledge at compile time of the type of "File object". That's the whole point... I (essentially) *need* an abstract interface, but I *want* the template functions for writing arrays and specific types and what not. Those 2 don't seem to be mixing very well.

struct BmpImage {  template < class Archive >  Image(Archive & archive) {    read(archive, width);    read(archive, height);    read(archive, pixels);  }}struct Loader {  virtual Image * loadImage();};struct FileLoader {  FileLoader(const char *filename) : f(filename) {}  virtual Image * loadImage() {    Image * img;    read(f, img);    return img;  }  File f;};

At some point, you then have an abstraction that provides loaders:

Loader * getLoaderBySource(const char * url) {    // if file    return new FileLoader(url);    // if http    return new HttpLoader(url);    // if foo    return new FooLoader(url);   }

and

template < class File >void read(File & f, Image * &result) {  int type;  read(f, type);  switch (type) {    case BMP : result = new BMPImage(f); break;    case GIF : result = new GIFImage(f); break;    default : result = null;  }}

And finally:

  Loader * loader = getLoaderBySource(url);  Image * x = loader.loadImage();}

Templates cannot resolve types at run-time, so they don't help there. Factory is used to abstract that away. This is also enforced through distinction between pointers and references when it comes to polymorphism.

Alternatively, instead of specializing templates and storing type in stream, you can also specifically specialize templates, and call appropriate version:

template < class File > void read(File & f, BMPImage & bmp);template < class File > void read(File & f, GIVImage & bmp);...switch (extension) {  case BMP : BMPImage b; read(f, b);  case GIF : GIFImage g; read(f, g);}

This approach is less viable, since images then need to share base class, or you need to use IOC approach.

Quote:
Aside from some flag and a switch/case for every function...

You can't. If data isn't known at compile-time, you need some form of factory. This one can then be implemented as factory, std::map, functors, ....

But at some point, you need to make cross-cut across all types and specify them manually. This is same with virtuals or templates. Templates merely allow you to avoid dynamic casts or guessing, so they are likely more efficient. They also don't require same base class.

rip-off

Feb 20, 2009 at 11:12 AM

Would this work:

class Image{public:	Image(std::istream &stream)	{		// use stream.read()	}};const char memData [] = { /* ... */ };int main(int argc, char **argv){	std::auto_ptr<std::istream> stream;	if(argc < 2)	{		std::stringstream *ptr = new std::stringstream();		ptr->write(memData, sizeof(memData));		stream.reset(ptr);	}	else 	{		stream.reset( new std::ifstream(argv[1]) );	}	Image image(*stream);}

The decisions here are made at runtime. And we can still have nice template functions to handle things like std::vector<>.

NotAYakk

Feb 20, 2009 at 12:05 PM

First, you should probably be passing in an output iterator to that function, not a container. That is a more standard way to fill a container. :) (and it works in more cases)

Second, if you want to pass that request onto a child class, you have to break it down binary-level steps. One solution is to provide an output interface that can handle +any+ type being written to it (the stream solution), then wrap your container/output iterator in that interface.

Another, more manual, option is to work out the steps that are done when you are doing a push back on a vector.

1> A type-dependant amount of data is read from your data source. Possibly it is validated.
2> A type-dependant amount of bits is provided by the vector.
3> A type-dependant operation is done, taking the data source information, and a pointer to the bits at the end of the vector storage, and a 'copy' is done.

...

If you assume the operation is going to be a copy construction, and the data source will be providing valid T's, and you are writing to T's in the vector...

You need to know how much information you need from the data stream. You need to know how to validate if that data is 'good'. You need to be able to 'roll back' if it isn't good.

You need to know how many bits you need from the data destination. If you are using a back insert iterator in your public interface, odds are you are going to construct it with a trivial constructor.

Then you'll use operator= to copy the source to the destination.

On a bit level, it might be:
1> Fetch sizeof(T) bytes from store
2> Check the bytes for validity, and roll back and fail if not.
3> Create a T() in the destination. Get a pointer to the T().
4> Pass a pointer to the source data to be copied, and the destination data to be copied. Call operator=(const T&) to move the data.

// pure binary interface:class copy_data_helper {public:  struct data_tag; // to aid in type safety.  Doesn't exist.  virtual size_t data_size_source() = 0;  virtual data_tag* get_destination_data() = 0;  virtual data_tag const* validate_source_data(void const*) = 0;  virtual void copy_data( data_tag const* src, data_tag* dest ) = 0;};// Implementation:template<typename T>class copy_data_impl_helper: public copy_data_helper{  static T* convert_tag( data_tag* t ){return reinterpret_cast<T*>(t);}  static T const* convert_tag( data_tag const* t ){return reinterpret_cast<T const*>(t);}  static data_tag* convert_tag( T* t ) {return reinterpret_cast<data_tag*>(t);}  static data_tag const* convert_tag( T* t ) {return reinterpret_cast<data_tag const*>(t);}public:  // should be implemented by child:  virtual T* get_destination_data_impl() = 0;  // can be reimplemented by child:  virtual T const* validate_source_data_impl(void const* input) {return reinterpret_cast<T const*>(input);}  virtual void copy_data_impl( T const* src, T* dest ){if (src&&dest) *dest = *src; }private: // implemented for you  virtual size_t data_size_source() { return sizeof(T); }  virtual data_tag*get_destination_data(){return convert_tag(get_destination_data_impl());}  virtual data_tag const*validate_source_data(void const*){/* you get the idea */}  virtual void copy_data( data_tag const* src, data_tag* dest ) {    return copy_data_impl(convert_tag(src), convert_tag(dest));  }};template<typename container>class copy_data_container: public copy_data_impl_helper<container::value_type> {  container& Container;public:  typedef container::value_type T;  virtual T* get_destination_data_impl() {    Container.push_back(T());    return &Container.back();  }};

There are some details missing, but the idea is you create a class that reduces the operations of adding something to a vector or other container to blind bit-level operations.

Then you write a class that partially implements the bit-level operations in a type-safe way.

Finally, each particular implementation implements the already type-safe details.

Your interface has a nice friendly WriteArray interface.

The WriteArray interface creates a copy_data_container< std::vector >, then passes a pointer/reference to this copy_data_helper to a pure virtual interface in your class, which then uses the copy_data_helper to do the work.

On the other side of the fence, you could even have the opposite of the copy_data_impl_helper that re-aliases the copy_data_helper into a type-safe construct. Hell, I could have added pure template methods to copy_data_helper that did the translation from bit-level to type-safe.

As noted, in this case it is probably a bad idea to reimplement the horse -- streams have solved this problem already. However, this trick of writing a generic interface that wraps a template-generated type-safe (or type-converting) implementation is useful. I especially find it useful when I mix it with the pImpl trick, which allows you to do this kind of type tricks while maintaining value semantics!

popsoftheyear

Feb 20, 2009 at 12:36 PM

I'm definitely missing something. I'm gonna have to read over your post a couple times NotAYakk, but everyone keeps saying use std::streams right? It's already been done, tested, approved, and stamped. I'm all for it! But I'm not seeing anything equivelant.

For example:

class SomeRandomClass {	int a;	std::string b;	float c;};std::fstream test;test.open("TestFile.bin");SomeRandomClass bla;test.write(&bla, 1);test << bla;

Now neither one of these works. But I can specialize the << operator for SomeRandomClass. In the case of my proposed interface, you can pass it any class whatsoever without having to specialize *every* *single* *class*.

Same goes for passing a vector. I would have 2 options:
1) Write a specialization (did I miss something that's already there?)
2) Use fstream::write which takes a const char *, losing all type safety of what the vector was.

Something must be going way over my head here. How have streams solved this aside from requiring one to write a specialization for every class they want to support?

-Scott

(patiently waiting on the "ahhhhhh I get it!" moment)

NotAYakk

Feb 20, 2009 at 1:31 PM

Let's back up a second then.

So you have some source of data. And you want to move it into a std::vector. All pretty reasonable.

Some questions:
1> Can this fail? If it fails, what should happen? Does failure relate to the particulars of type T? In what way?
2> Do you want virtual behaviour (or dispatch behavior) in the type T cross implementation of store case? Or is virtual behaviour in the type of T (compile time virtual behaviour, which is not so virtual, heh), and distinct virtual behaviour on the part of the implementation of the data source?

If you want to cross compile-time polymorphism (templates) with run-time polymorphism, you need to explicitly list all of the types T that should have virtual behaviour.

This can be done like this:

class Helper {  // Help for dealing with a type on the 'raw binary' level.};template<typename T>class HelperImpl: public Helper {  // Moves between the template type-safe and the Helper bit-based world  // in a generated way.  Implements the Helper interface.};class SomeBaseClass{public:  void ByteBased( Helper* help, void* raw, size_t size );  template<typename T>  void DoStuffHelper( T*t ) {    HelperImpl<T> helper;    ByteBased( &helper, (void*)t, sizeof(T) );  };  // each type you ever want to virtually override goes here:  // (these are overrides, not template specializations)  virtual void DoStuff( int* dest ) { DoStuffHelper( dest ); }  virtual void DoStuff( double* dest ) { DoStuffHelper( dest ); }  // This catches everything you don't override:  template<typename T>  void DoStuff( T* dest ) { DoStuffHelper( dest ); }};

This allows doubly-'virtual' behavior between compile-time T and run-time SomeBaseClass implementations. Each 'type of T' x 'implementation of SomeBaseClass' dynamic dispatch behavior option needs to be added to SomeBaseClass, and have it auto-forward to the 'default' implementation (which ends up doing a byte-based action).

Pure compile-time virtual behavior can be implemented in HelperImpl via specialisation.

Pure run-time virtual behavior can be implemented in ByteBased(...) detail.

Run x Compile-time virtual behavior requires an added DoStuff virtual function to be added to SomeBaseClass, then have it be overridden in the implementation of your SomeBaseClass.

And yes, inside the implementation of SomeBaseClass, you end up having to do byte-based operations with help from the Helper. You can make the operations as 'safe as they can be' by making the Helper smarter (so it knows how many bytes you need to feed it, does the in-place news for you, it can validate the data, etc). It can even be responsible for managing the vector's memory for you in a type-safe manner.

Now you can go even stranger, and use typeids and/or build tables and the like. But no, you cannot get 'pretty' virtual template behaviour.

An implementation of an interface needs to be able to be compiled down to assembly without ever knowing how the interface is used. Languages that compile to 'high level abstractions', or don't compile at all, can pull off additional tricks, where how the interface is used causes the back-end to be compiled at that moment.

Bregma

Feb 20, 2009 at 1:45 PM

Quote:
Original post by popsoftheyear
Now neither one of these works. But I can specialize the << operator for SomeRandomClass. In the case of my proposed interface, you can pass it any class whatsoever without having to specialize *every* *single* *class*.

I'm not sure what you're trying to do but I can't see how you're going to be able to have class-specific functionaly without writing it.

Now, as for using operator<<() on an str stream, it's already defined as formatting an internal value for external (text) storage and vice versa for operator>>(). If what you want to do is save binary data you will want to either overload another operator or use a plain old overloaded function.

For example (off the top of my head, untested):

class SomeRandomClass {	int a;	std::string b;	float c;};std::ostream& save(std::ostream& ostr, const SomeRandomClass& d){  const std::string::size_type s = d.b.size();  ostr.write(reinterpret_cast<const char*>(&d.a), sizeof(d.a));  ostr.write(reinterpret_cast<const char*>(&s), sizeof(std::string::size_type));  ostr.write(d.b.c_str(), d.b.size());  ostr.write(reinterpret_cast<const char*>(&d.c), sizeof(d.c));}int main(){  std::ofstream test("TestFile.bin");  if (test)  {    SomeRandomClass bla;    save(test, bla);  }  SomeRandomClass blik[120];  std::foreach(blik, blik+(sizeof(blik)/sizeof(SomeRandomClass), binder_1st(test, save));}

popsoftheyear

Feb 20, 2009 at 2:07 PM

Right... it just occurred to me that I'm looking at this all from a perspective that any arbitrary class can be treated as POD and that's just wrong. We're dealing with a lot of old code and structures that are POD so it never occurred to me otherwise. I'm pretty sure this was my stumbling block... so... thanks to everyone for their input. Feel free to add anything else you think of. Thanks again.

Cheers
-Scott

Moomin

Feb 20, 2009 at 2:45 PM

2 things

1) Non-POD classes are going to need specialization for each class for serialization
2) std::istream, not necessarily std:fstream

NotAYakk

Feb 22, 2009 at 11:26 AM

Quote:
Original post by Moomin
2 things

1) Non-POD classes are going to need specialization for each class for serialization

Note you can do some tricks to teach your non-POD classes how to serialize with less overhead.

First, you abstract out the serialization method to a traits class, so the rest of your template class doesn't have to be rewritten by the specializer.

Second, you can use tags and other tricks to "auto-detect" serialization interfaces. Either how the STL specializes std::advance (tags), or you can have conventions (if you have a type called "can_be_serialized", then the Serialize( char*, size_t* ) method exists).

In C++09 you can use concepts to do a duck-type check on the passed in parameters, and very cleanly search for various conventional interfaces.

The downside is that there is no way to say "if data_is_pod" and "if data_is_flat" that I'm aware of. :( So if you have a default, POD-like, implementation, you have to blindly claim it is true, and have horribly evil behavior happen if you are wrong.

phresnel

Feb 24, 2009 at 6:56 AM

Quote:
Original post by NotAYakk
The downside is that there is no way to say "if data_is_pod" and "if data_is_flat" that I'm aware of. :( So if you have a default, POD-like, implementation, you have to blindly claim it is true, and have horribly evil behavior happen if you are wrong.

#include <iostream>#include <tr1/type_traits>struct Pooh {};struct Mickey { Mickey (); };int main () {    using namespace std::tr1;    std::cout << (is_pod<Pooh>::value ? "Pooh is POD.\n" : "Pooh is Non-POD.\n");    std::cout << (is_pod<Mickey>::value ? "Mickey is POD.\n" : "Mickey is Non-POD.\n");}

$ g++-4.1 -std=c++0x is_pod.cc && ./a.out cc1plus: error: unrecognized command line option "-std=c++0x"$ g++-4.1 is_pod.cc && ./a.out Pooh is Non-POD.Mickey is Non-POD.

$ g++-4.2 -std=c++0x is_pod.cc && ./a.out cc1plus: error: unrecognized command line option "-std=c++0x"$ g++-4.2 is_pod.cc && ./a.out Pooh is Non-POD.Mickey is Non-POD.

$ g++-4.3 -std=c++0x is_pod.cc && ./a.out Pooh is POD.Mickey is Non-POD.

[...]namespace std { namespace tr1 {    template<> struct is_pod <Pooh> : public true_type { };} }[...]

$ g++-4.1 is_pod.cc && ./a.out Pooh is POD.Mickey is Non-POD.

The moral: In C++0x, there is (also available in TR1 as , under GNU), and in , there is std::is_pod. But if your compiler doesn't support some builtin functionailty w.r.t. is_pod<>, then you have to inject you own specialisation of is_pod<> into namespace std. As you can see, g++-4.3 together with -std=c++0x has such builtin functionality, for older versions, you have to inject your own std::is_pod<> specialization.

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.